Vehicle recording device

Abstract

To provide a vehicle recording device which can suppress the storage data size in such a configuration that periodically stores image data captured by a camera.SOLUTION: A processor in a camera ECU acquires image frames captured by a camera at a prescribed sampling interval, and cuts out as a partial image PI a portion that is set as a cut-out region CR in each image frame. The partial image is, for example, cut out in a landscape format so as to include a prescribed line range in the image frame. The cut-out region CR is set in a portion in which an arbitrary target object such as a road surface, the sky, and a preceding vehicle is likely to be captured in the image frame. The processor generates a connected image in which the plurality of partial images with different imaging time points (acquired time points) are connected in a direction orthogonal to a cut-out direction and stores the connected image in a storage memory.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present disclosure relates to a vehicle recording device that stores image data indicating the situation outside / inside the vehicle.

Background Art

[0002] Patent Document 1 discloses a recording device that continuously records camera images in a buffer memory or the like and stores the video data for a predetermined time before and after the detection of a predetermined incident in an external memory. This type of recording device is also referred to as an event recording type drive recorder. In addition, as a device for recording the situation outside the vehicle other than the event recording type, there is also a continuous recording type (always recording type) drive recorder that continuously records camera images on a storage medium.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In an event recording type drive recorder, there is a possibility that the object of interest, which is what the viewer of the video wants to confirm, may not be recorded partially / entirely. According to the always recording type drive recorder, the possibility that the object of interest is not recorded can be reduced, but the size of the stored data may increase.

[0005] The present disclosure has been made based on the above considerations, and one of its purposes is to provide a vehicle recording device capable of suppressing the size of stored data in a configuration where the image data captured by a camera is periodically stored.

Means for Solving the Problems

[0006] The vehicle recording device disclosed herein includes: a video acquisition unit (F1) that acquires video signals from a camera installed in the vehicle; an image acquisition unit (F41) that sequentially acquires image frames generated based on the video signals acquired by the video acquisition unit at predetermined sampling intervals; an extraction unit (F42) that extracts a row range or column range set as an extraction region as a partial image (PI) from the image frame acquired by the image acquisition unit; a concatenation processing unit (F43) that concatenates the multiple partial images extracted by the extraction unit in chronological order in a direction orthogonal to the extraction direction, which is the direction in which the partial images are extracted from the image frame; and a concatenated image (CI) which is an image formed by the concatenation processing unit by which the multiple partial images are connected. to place It comprises a storage processing unit (F44) that stores the data in a designated storage medium (15), The vehicle recording device is configured such that the extraction unit vertically extracts an image frame as a partial image, including a predetermined column range within the image frame, and the merging unit horizontally merges the partial images. Other vehicle recording devices disclosed include: a video acquisition unit (F1) that acquires video signals from a camera installed in the vehicle; an image acquisition unit (F41) that sequentially acquires image frames generated based on the video signals acquired by the video acquisition unit at predetermined sampling intervals; an extraction unit (F42) that extracts a row range or column range set as an extraction region as a partial image (PI) from the image frames acquired by the image acquisition unit; a concatenation processing unit (F43) that concatenates multiple partial images extracted by the extraction unit in chronological order in a direction orthogonal to the extraction direction, which is the direction in which the partial images are extracted from the image frame; and an image generated by the concatenation processing unit, which is a sequence of multiple partial images. A vehicle recording device comprising: a storage processing unit (F44) that stores a concatenated image (CI) in a predetermined storage medium (15), the device further comprising: a recognition processing unit (F45) that recognizes an object from an image formed by a video signal; and an image processing unit (F46) that generates an image in which a marker (Mk), which is an image indicating the recognition state of the object, is superimposed on the image formed by the video signal, the image acquisition unit acquires the image with the superimposed marker as an image frame, and the cropping unit is configured to crop a region including a part or all of the marker as a partial image if the marker is placed on the cropping region of the image frame acquired by the image acquisition unit. Other vehicle recording devices disclosed include: a video acquisition unit (F1) that acquires video signals from a camera installed in a vehicle; an image acquisition unit (F41) that sequentially acquires image frames generated based on the video signals acquired by the video acquisition unit at predetermined sampling intervals; an extraction unit (F42) that extracts a row range or column range set as an extraction region as a partial image (PI) from the image frames acquired by the image acquisition unit; a concatenation processing unit (F43) that concatenates a plurality of partial images extracted by the extraction unit in chronological order in a direction orthogonal to the extraction direction, which is the direction in which partial images are extracted from the image frame; and a storage processing unit (F44) that saves a concatenated image (CI), which is an image consisting of a plurality of partial images connected together and generated by the concatenation processing unit, to a predetermined storage medium (15). The vehicle recording device further includes an image processing unit (F46) that generates a processed image, which is an image obtained by applying predetermined processing to the image shown by the video signal. The image acquisition unit is configured to acquire the processed image as an image frame at sampling intervals, and the extraction unit is configured to extract a predetermined range of the processed image as a partial image. Other vehicle recording devices disclosed include: a video acquisition unit (F1) that acquires video signals from a camera installed in the vehicle; an image acquisition unit (F41) that sequentially acquires image frames generated based on the video signals acquired by the video acquisition unit at predetermined sampling intervals; an extraction unit (F42) that extracts a row range or column range set as an extraction region as a partial image (PI) from the image frame acquired by the image acquisition unit; a concatenation processing unit (F43) that concatenates multiple partial images extracted by the extraction unit in chronological order in a direction orthogonal to the extraction direction, which is the direction in which the partial images are extracted from the image frame; and an image generated by the concatenation processing unit, which is a sequence of multiple partial images. A vehicle recording device comprising a storage processing unit (F44) that stores a concatenated image (CI), which is an image, in a predetermined storage medium (15), wherein the video acquisition unit acquires video signals from each of a plurality of cameras installed in the vehicle, the image acquisition unit acquires image frames for each camera at sampling intervals, the extraction unit generates partial images for each camera, and the concatenation processing unit is configured to generate a camera integrated partial image (CPI) by combining the partial images for each camera that are captured at the same time in a predetermined order in the extraction direction, and to generate a concatenated image by combining the camera integrated partial images in a time-series order in a direction orthogonal to the extraction direction. Other vehicle recording devices disclosed include: a video acquisition unit (F1) that acquires video signals from a camera installed in the vehicle; an image acquisition unit (F41) that sequentially acquires image frames generated based on the video signals acquired by the video acquisition unit at predetermined sampling intervals; an extraction unit (F42) that extracts a row range or column range set as an extraction region as a partial image (PI) from the image frame acquired by the image acquisition unit; a concatenation processing unit (F43) that concatenates a plurality of partial images extracted by the extraction unit in chronological order in a direction orthogonal to the extraction direction, which is the direction in which the partial images are extracted from the image frame; and a storage processing unit (F44) that saves a concatenated image (CI), which is an image of a plurality of partial images connected together and generated by the concatenation processing unit, to a predetermined storage medium (15); wherein the camera is mounted so as to include in its imaging range a direction that is 10° or more upward from the vehicle horizontal plane (RP), which is a plane orthogonal to the height direction of the vehicle; and the extraction region is set to an area in the image frame in which the sky can be captured.

[0007] According to the above configuration, instead of saving the image frames acquired sequentially at the sampling interval in their original size, only a portion corresponding to the cropped area is extracted and saved as a concatenated image arranged in chronological order. Therefore, the data size can be reduced.

[0008] The reference numerals in parentheses in the claims indicate the correspondence with the specific means described later in the embodiments, and do not limit the technical scope of this disclosure. [Brief explanation of the drawing]

[0009] [Figure 1] This is a block diagram showing the overall structure of the camera image processing system. [Figure 2] This figure shows an example of the camera's mounting position and orientation. [Figure 3] This is a functional block diagram of the camera ECU. [Figure 4] This is a diagram illustrating the operation of the region of interest extraction unit. [Figure 5] This is a diagram for explaining the operation of the data processing unit. [Figure 6] This is a diagram showing the configuration of the concatenated image. [Figure 7] This is a flowchart for explaining the operation of the data processing unit related to the generation of the concatenated image. [Figure 8] This is a diagram showing an example of the setting of the cut-out area. [Figure 9] This is a diagram showing an example of the concatenated image generated by the setting of the cut-out area shown in FIG. 8. [Figure 10] This is a diagram showing another example of the setting of the cut-out area. [Figure 11] This is a diagram showing the concatenated image generated by the setting of the cut-out area shown in FIG. 10. [Figure 12] This is a diagram showing another example of the setting of the cut-out area. [Figure 13] This is a diagram showing the concatenated image generated by the setting of the cut-out area shown in FIG. 12. [Figure 14] This is a block diagram showing a modified example of the configuration of the data processing unit. [Figure 15] This is a diagram schematically showing the sequence in the case of generating a concatenated image based on an image frame including a marker indicating a recognition result. [Figure 16] This is a diagram showing a group of image frames including a marker indicating a recognition result. [Figure 17] This is a diagram showing an example of the concatenated image generated based on the group of image frames shown in FIG. 16. [Figure 18] This is a diagram for explaining a configuration in which the cut-out width is dynamically adjusted according to the recognition status of the target. [Figure 19] This is a block diagram showing a configuration including a plurality of cameras. [Figure 20] This is a diagram for explaining a configuration in which images of a plurality of cameras are collectively stored. [Figure 21] This is a diagram for explaining another example of the configuration in which images of a plurality of cameras are collectively stored. [Figure 22] This is a diagram for explaining a configuration in which a partial image is extracted based on an aerial image. [Figure 23] This diagram illustrates how sampling intervals and other parameters are changed depending on the scene. [Figure 24] This block diagram shows other configuration examples of camera image processing systems. [Figure 25] This is a functional block diagram showing a case where the data processing unit includes an inspection processing unit. [Figure 26] This figure shows an example of setting the cutting area when an overhead marker is set on the object to be inspected. [Figure 27] This figure shows examples of items included in the shooting conditions data. [Figure 28] This is a diagram illustrating the operation of the inspection and processing unit. [Figure 29] This is a flowchart illustrating the operation of the camera ECU during image diagnostic processing. [Figure 30] This flowchart shows other examples of camera ECU operation related to image diagnostic processing. [Modes for carrying out the invention]

[0010] The embodiments of this disclosure will be described below with reference to the figures.

[0011] <Introduction> The camera image processing system Sys relating to this disclosure is used when mounted on a vehicle. In the following description, "the vehicle" refers to the vehicle on which the camera image processing system Sys is mounted. In the following description, the front-rear, left-right, and up-down directions are defined relative to the vehicle. Specifically, the front-rear direction corresponds to the longitudinal direction of the vehicle. The left-right direction corresponds to the width direction of the vehicle. The up-down direction corresponds to the vehicle height direction. From another perspective, the up-down direction corresponds to the direction perpendicular to the plane parallel to the front-rear and left-right directions. In this disclosure, the plane perpendicular to the vehicle height direction is also referred to as the vehicle horizontal plane RP. The vehicle horizontal plane RP corresponds to the horizontal plane determined relative to the vehicle.

[0012] In this disclosure, "parallel" does not mean perfectly parallel. It may mean tilted by a few degrees to about 20 degrees. In other words, it can include a state that is roughly parallel (a so-called approximate parallel state). Similarly, the expression "perpendicular" in this disclosure does not mean perfectly perpendicular, but can also include a state that is tilted by a few degrees to about 20 degrees.

[0013] <Explanation of the overall structure> Figure 1 is a diagram showing an example of the schematic configuration of the camera image processing system Sys according to this disclosure. As shown in Figure 1, the camera image processing system Sys comprises a camera ECU 1, a camera 2, an input device 3, an in-vehicle sensor 4, and an in-vehicle ECU 5. In the component names, ECU is an abbreviation for Electronic Control Unit, meaning an electronic control unit.

[0014] Camera ECU1 is connected to Camera 2, Input Device 3, Onboard Sensor 4, and Onboard ECU 5 in a communication-enabled manner. Camera ECU1 is connected to the various devices / sensors mentioned above via an in-vehicle network, which is a communication network built within the vehicle. Some of the various devices / sensors mentioned above may be individually connected to Camera ECU1 by dedicated lines. For example, Camera 2 and Camera ECU1 may be directly connected by a dedicated video signal line.

[0015] Camera ECU1 is an ECU that performs various processes based on the video signal input from Camera 2. Camera ECU1 also functions as a vehicle recording device. Specifically, Camera ECU1 performs the process of saving a specific area of ​​an image frame, extracted at a predetermined sampling rate from the temporally continuous image frames that constitute the video signal input from Camera 2, to a predetermined memory. The output of Camera 2 is not limited to video; it may also be a still image. The term "video signal" includes image signals.

[0016] In a more preferred configuration, the camera ECU1 can recognize objects designated as detection targets based on the video signal input from camera 2. Furthermore, the camera ECU1 can perform processing to display the video captured by camera 2 on a display, either directly or after predetermined processing. For example, it can generate and display on any display images such as a bird's-eye view image obtained by changing the viewpoint of the camera image, or an image with a marker image indicating the recognition result superimposed on the camera image. Details of the camera ECU1's functions will be described separately.

[0017] The camera ECU1 is implemented using a computer. Specifically, the camera ECU1 includes a processor 11, RAM (Random Access Memory) 12, storage 13, I / O 14, storage memory 15, and bus lines connecting these components. The processor 11 is hardware for arithmetic processing (in other words, an arithmetic core) coupled with the RAM 12. The processor 11 is, for example, a CPU (Central Processing Unit). The processor 11 may also be implemented using a GPU (Graphics Processing Unit). The processor 11 performs various processes corresponding to each functional unit described later by accessing the RAM 12. The RAM 12 is a volatile storage medium.

[0018] Storage 13 is a non-volatile storage medium such as flash memory. Storage 13 is located on the circuit board that constitutes the camera ECU1 together with the processor 11 and RAM 12, and corresponds to so-called internal storage. Storage 13 stores dictionary data / identification models used for image recognition processing, drawing data for generating composite images, etc. I / O 14 is a circuit module for communicating with other devices. I / O 14 corresponds to an input / output circuit. I / O 14 is implemented using analog circuit elements or ICs.

[0019] The storage memory 15 is a rewritable, non-volatile storage medium. The storage memory 15 is a removable storage medium, such as an SD memory card (a so-called SD card). The storage memory 15 corresponds to the storage location of concatenated image data based on camera footage. The storage memory 15 may also be a memory device incorporated into a circuit board, similar to the storage 13.

[0020] Camera 2 comprises at least a lens and an image sensor, and electronically acquires an image showing the area around the vehicle. Camera 2 is mounted in a predetermined position and orientation on the vehicle to capture a predetermined range outside the vehicle. In this embodiment, camera 2 is, for example, an on-board camera that captures the area in front of the vehicle. As shown in Figure 2, camera 2 is positioned at the upper end of the windshield on the interior side of the vehicle. Such a camera 2 may be called a front camera. Note that camera 2 as a front camera may also be positioned on the front grille or front bumper.

[0021] Camera 2 has a horizontal field of view of approximately 60° to 130° and a vertical field of view of approximately 45° to 60°. Of course, Camera 2 may be a wide-angle camera using a fisheye lens. Alternatively, Camera 2 may be a narrow-angle / telephoto camera with a horizontal field of view of less than 30°. Camera 2 is mounted in a position where its optical axis is parallel to the vehicle's horizontal plane RP, or angled a few degrees downward / upward from the vehicle's horizontal plane RP. Camera 2 may also be mounted in a position angled downward or upward by approximately 10°. In this embodiment, Camera 2 is mounted in a position that allows it to image the road surface 10m or more ahead, and also to image the sky in front. With this mounting position, the image frame captured by Camera 2 may include the road surface and the sky above.

[0022] In Figure 2, θu represents the upper limit angle, which is the angle that the upper boundary Bu of the area that camera 2 can image in the vertical direction makes with respect to the vehicle's horizontal plane RP. Similarly, θd represents the lower limit angle, which is the angle that the lower boundary Bd of the area that camera 2 can image in the vertical direction makes with respect to the vehicle's horizontal plane RP. The sum of the upper limit angle θu and the lower limit angle θd corresponds to the vertical field of view. The bisector of the interior angle between the upper boundary Bu and the lower boundary Bd corresponds to the optical axis. Camera 2 is designed with an upper limit angle θu of 10° or more to enable imaging of the sky. Camera 2 is also designed with a lower limit angle θd of 20° or more to enable imaging of the road surface. Such a mounting position for camera 2 corresponds to a position where the imaging range includes a direction that is 10° or more upward from the vehicle's horizontal plane. Furthermore, the above mounting position corresponds to a position where the imaging range includes a direction that is 20° or more downward from the vehicle's horizontal plane.

[0023] Camera 2 outputs a video signal based on the captured image to Camera ECU 1. The frame rate of Camera 2 is set to, for example, 30fps (frames per second). Of course, the frame rate of Camera 2 may also be 60fps or other values. The frame rate of Camera 2 may be configured to be dynamically changed based on instruction signals from Camera ECU 1. The video signal output by Camera 2 corresponds to a set of temporally consecutive image frames. The size of the image frame is, for example, 1920 x 1080 pixels. Of course, the size of the image frame may also be 2880 x 1860 pixels or other values.

[0024] Input device 3 is a device that receives user instructions for camera ECU 1. Input device 3 is a switch for instructing camera ECU 1 to start / stop recording, etc. Input device 3 is, for example, installed on the steering wheel of the vehicle as one of the steering switches. Input device 3 outputs an operation signal to camera ECU 1 indicating the content of the operation performed by the user. Input device 3 may be a hardware switch installed on the instrument panel, or a touch panel stacked on a display. The user here may be the occupant in the driver's seat (i.e., the driver), or the occupant in the passenger seat.

[0025] The on-board sensor 4 is a sensor that detects state variables related to the vehicle's driving control. The on-board sensor 4 includes a vehicle speed sensor, steering angle sensor, acceleration sensor, and yaw rate sensor. The vehicle speed sensor detects the vehicle's driving speed. The steering angle sensor detects the steering angle (the so-called steering angle). The acceleration sensor detects acceleration acting on the vehicle in at least one of the following directions: longitudinal, lateral, and vertical. Here, a 3-axis acceleration sensor is assumed to be used as the acceleration sensor. The detected value of the acceleration sensor can be used as material for determining the vehicle's attitude relative to the horizontal plane. The yaw rate sensor detects the yaw rate acting on the vehicle. Note that the types of sensors used by the camera image processing system Sys as on-board sensor 4 can be designed as appropriate, and it is not necessary to have all of the above-mentioned sensors. In addition, a compass sensor, rain sensor, illuminance sensor, GNSS (Global Navigation Satellite System) receiver, and collision detection sensor can also be included in the on-board sensor 4. Each sensor outputs data to the camera ECU1 indicating the current value (i.e., the detection result) of the physical state variable it is trying to detect.

[0026] The compass sensor is a sensor that detects the azimuth angle corresponding to the direction of travel of the vehicle. A geomagnetic sensor can be used as the compass sensor. The rain sensor is a sensor that detects rainfall. The illuminance sensor is a sensor that detects the brightness (illuminance) outside the vehicle. The GNSS receiver is a device that receives navigation signals transmitted from positioning satellites that make up the GNSS and detects the position coordinates of the GNSS receiver sequentially (for example, every 100 milliseconds). As for the GNSS, GPS (Global Positioning System), GLONASS, Galileo, IRNSS, QZSS, Beidou, etc. can be used.

[0027] The in-vehicle ECU 5 is an ECU other than the camera ECU 1. The in-vehicle ECU 5 is, for example, a communication ECU or a driver assistance ECU. The communication ECU is an ECU that controls wireless communication between the vehicle and the outside world. The communication ECU may be equipped with a communication module for performing cellular communication such as 4G or 5G, or a communication module for performing Wi-Fi® communication. The communication ECU may perform the process of acquiring data stored in the storage memory 15 from the camera ECU 1 and transferring it to a predetermined server or user terminal. The transfer of stored data by the communication ECU may be performed based on user operation, or it may be performed automatically based on the fulfillment of predetermined transfer conditions. Transfer conditions may include, for example, turning off the driving power supply, the free capacity of the storage memory 15 falling below a predetermined value, or detecting a collision.

[0028] The driver assistance ECU is an ECU that performs processes to assist the driver's driving operations. Based on instructions from the driver to perform control, the driving ECU can automatically perform some or all of the driving control, such as acceleration, deceleration, and steering, on behalf of the driver. The driver assistance ECU may also have the function of an automated driving system, which performs autonomous driving. Based on the recognition results of surrounding objects by the camera ECU 1, the driver assistance ECU may notify the driver of collisions with other moving or stationary objects. Other moving objects include pedestrians, other vehicles, cyclists, etc. For example, during low-speed driving such as when turning right or left at an intersection, when exiting a parking space, or when parking, the presence of pedestrians or other objects that may be involved in a collision may be notified to the driver by sound output from the speaker or image display on the display. The driver assistance ECU may also have the function of a parking assistance ECU, which assists / automatically performs driving control for parking.

[0029] <About the camera ECU functions> The camera ECU1 includes various functional units as shown in Figure 3. Specifically, it includes an image acquisition unit F1, an operation reception unit F2, a vehicle status acquisition unit F3, and a data processing unit F4. Some or all of these functional units can be manifested by the processor 11 executing a program stored in the storage 13. Of course, some of these functional units may also be implemented using hardware such as an IC (Integrated Circuit).

[0030] The video acquisition unit F1 acquires a video signal from the camera 2. The video acquisition unit F1 converts the video signal input from the camera 2 into digital image data in a predetermined data format and outputs it to the data processing unit F4. The operation reception unit F2 receives user operations such as starting and stopping image recording based on the input signal from the input device 3. The operation reception unit F2 may also be configured to receive instruction operations to transfer data stored in the storage memory 15 to a predetermined device based on the signal from the input device 3.

[0031] The vehicle state acquisition unit F3 acquires vehicle information from the on-board sensor 4, which is information indicating the behavior and state of the vehicle, such as vehicle speed, steering angle, and yaw rate. The vehicle information also includes the operating status of the turn signals, the operating status of the wipers, and the shift position. In a more preferred embodiment, the vehicle state acquisition unit F3 may detect points of change (transitions) in the driving scene, such as turning left or right, stopping, starting, and parking, based on the input from the on-board sensor 4 and the image recognition results. Points of change in the driving scene can be used as triggers to create a new dataset to be used as the target for concatenation of partial image PIs.

[0032] The data processing unit F4 is a module that performs various processing using camera images input from the video acquisition unit F1. The data processing unit F4 includes, as sub-function units, a base image acquisition unit F41, a region of interest extraction unit F42, a concatenation processing unit F43, a storage processing unit F44, and a recognition processing unit F45. The processor 11 / circuit module functioning as the data processing unit F4 corresponds to a vehicle recording device.

[0033] The base image acquisition unit F41 extracts image frames from the video data acquired by the video acquisition unit F1 at predetermined sampling intervals to be used as base image data. The base image is an image used by the region of interest extraction unit F42, etc., and refers to the data that will be used as the basis for the partial image PI and concatenated image CI described later. The base image can be understood as an image for recording, or an image for generating concatenated images / partial images. The process of extracting image frames from video data at sampling intervals can be understood in one aspect as a downsampling or decimation process.

[0034] The sampling interval is set to a length that is necessary and sufficient to check the state of a given object of interest, such as 1 second, 2 seconds, or 10 seconds. Of course, depending on the type / characteristics of the object of interest, the sampling interval can be set to a value smaller than 1 second, such as 100 milliseconds, 250 milliseconds, or 500 seconds. The sampling interval can be set to M times the imaging period of camera 2. M is a parameter corresponding to the downsampling rate. M is an integer of 2 or more, and is set to, for example, 5, 10, or 25. The shooting period is the interval at which images are taken, and corresponds to the reciprocal of the frame rate. The shooting period corresponding to 30fps is approximately 33 milliseconds.

[0035] Possible subjects of interest include, for example, the condition of the sky (i.e., weather conditions) and road surface conditions. Weather does not change rapidly in short periods of time, such as one second. Therefore, if weather conditions are the subject of interest, the sampling interval may be set to 2 seconds, 4 seconds, or 10 seconds. In other words, the downsampling rate M may be set to 100, 200, or 500.

[0036] Furthermore, if the object of interest is road surface conditions, more specifically weather-related factors such as the presence or absence of snow, puddles, or sand cover, the sampling interval may be set to 1 second or 2 seconds. Additionally, if the object of interest is the presence or absence of a preceding vehicle, or the distance to a preceding vehicle, the sampling interval may be set to 500 milliseconds, 1 second, or 2 seconds.

[0037] When the object of interest is the recognition of lane markings, the sampling interval can be set to 200 milliseconds, 400 milliseconds, or 1 second. Similarly, when the object of interest is the recognition of objects within a predetermined distance from the vehicle, such as preceding vehicles or pedestrians, the sampling interval can also be set to 200 milliseconds, 400 milliseconds, or 1 second. The sampling interval may also be dynamically adjusted according to the vehicle's speed. The sampling interval may be set shorter as the speed increases.

[0038] The region of interest extraction unit F42 extracts a region pre-set as the extraction region CR from the recording image frame acquired by the base image acquisition unit F41 as a partial image PI. As shown in Figure 4, the extraction of the partial image PI can be selectively performed in either the vertical or horizontal direction. Figure 4(A) shows the method of extracting the partial image PI in the horizontal direction, and (B) shows the method of extracting the partial image PI in the vertical direction. The arrows in Figure 4 indicate the extraction direction, and the dashed lines conceptually show the region that is actually extracted. The partial region is extracted such that the length in either the horizontal or vertical direction is the same as the original frame.

[0039] The cropping direction is selected considering the range in which the subject of interest will be captured. The cropping direction is basically horizontal. However, the cropping direction may be vertical depending on the purpose / subject of interest. The cropping direction and cropping area can be specified by the operator via the recording settings screen. The recording settings screen is a screen for changing the specifications related to recording and is displayed on the in-vehicle display, etc., based on the signal from input device 3. The cropping direction is constant in a single recording process.

[0040] Extracting a partial image PI horizontally from an image frame is equivalent to extracting an image row by row. One row in an image frame corresponds to a group of pixels arranged in one column horizontally. Extracting row by row also includes extracting multiple rows at once. When extracting a partial image PI horizontally, the horizontal length of the partial image PI matches the horizontal length of the original image. When extracting a partial image PI horizontally, the vertical length of the partial image PI can be kept constant. When extracting a partial image PI horizontally, the extraction width Wc hereafter refers to the vertical length of the partial image PI / extracted region CR.

[0041] When the cropping direction is horizontal, the cropping area CR is represented by the row range to be cropped. That is, the cropping area CR can be defined by a row number indicating the cropping start position and a row number indicating the cropping end position. For example, the cropping width can be set to a desired size that allows the object of interest to be recorded, such as 150 pixels (px), 200px, or 400px. When cropping a partial image PI horizontally, the cropping width Wc may be defined based on the frame length, which is the length of the image frame in the vertical direction. For example, the cropping width Wc can be set to 5%, 10%, or 15% of the vertical length. Alternatively, the cropping width Wc may be dynamically adjusted according to the driving scene or the recognition state of a predetermined object. The cropping width Wc can be expressed in pixels.

[0042] Furthermore, extracting a partial image PI vertically from the original image frame is equivalent to extracting an image column by column. One column in an image frame corresponds to a group of pixels arranged in a vertical column. Note that extracting column by column also includes extracting multiple columns at once. When extracting a partial image PI vertically, the extraction width Wc corresponds to the horizontal length of the partial image PI. The vertical length of the partial image PI when extracted vertically matches the vertical length of the original image. When extracting a partial image PI vertically, the extraction width Wc hereafter refers to the horizontal length of the partial image PI / extracted region CR.

[0043] The cropping width Wc when a partial image PI is cropped vertically can also be kept constant. When the cropping direction is vertical, the cropping region CR is represented by the range of columns to be cropped. That is, the cropping region CR can be defined by a column number indicating the cropping start position and a column number indicating the cropping end position. Alternatively, the horizontal length may be adjusted dynamically. The cropping width Wc when a partial image PI is cropped vertically may be dynamically determined based on the frame width, which is the length of the image frame in the horizontal direction. For example, the cropping width Wc may be set to 5%, 10%, 15%, etc., of the frame width.

[0044] The cropping area CR is set to encompass the area where the object of interest is expected to be captured. For example, if the object of interest is the sky, the cropping area CR is set above the center of the image frame. If the object of interest is the road surface, the cropping area CR is set below the center of the image frame. The cropping area CR may be changed depending on the mounting orientation of camera 2. The cropping area CR can be called the area of ​​interest or area of ​​focus because it corresponds to the image area of ​​interest of the person reviewing the recorded data. The cropping width Wc may be set so that the total amount of data recorded in a given time does not exceed the capacity of the storage memory 15.

[0045] The concatenation processing unit F43 generates a concatenated image CI by combining the partial images PI extracted by the region of interest extraction unit F42 in a direction perpendicular to the extraction direction. In this disclosure, the direction perpendicular to the extraction direction is also referred to as the concatenation direction. For example, as shown in Figure 5, if the extraction direction is horizontal, the extracted partial images PI are concatenated vertically in chronological order. The concatenation length Lc, which is the length of the concatenated image CI in the concatenation direction, is updated each time a new partial image PI is extracted.

[0046] The concatenation processing unit F43 creates a new concatenated image CI starting with the next provided partial image PI, based on the fact that the number of concatenations, which is the number of partial image PIs constituting the concatenated image CI, exceeds a predetermined value. The concatenation processing unit F43 may also create a new concatenated image CI based on the fact that the concatenation length Lc exceeds a predetermined value. Generating a new concatenated image CI is equivalent to saving the subsequently extracted partial image PIs as a separate file (dataset). A timestamp indicating the time of capture may be attached to the corner of each partial image PI constituting the concatenated image CI, as shown in Figure 6. The position of the timestamp within the partial image PI may be configured to be specified by the operator via the recording settings screen. Such a configuration can reduce the risk of the timestamp overlapping with the object of interest. Note that the time of capture information for each partial image PI may be attached to the image itself as metadata.

[0047] The storage processing unit F44 performs the process of periodically saving the linked image CI generated by the linking processing unit F43 to the storage memory 15 at predetermined intervals. For example, the linking processing unit F43 saves the linked image CI to the storage memory 15 when the number of linked parts or the linked length Lc exceeds a predetermined value. The storage processing unit F44 may also save linked image CI generated during a trip to the storage memory 15 based on the fact that the driving power supply has been turned off. The driving power supply here refers to the power supply used to operate the drive source / the power supply that turns on in conjunction with the operation of the drive source. For example, if the vehicle is an engine-powered vehicle, the ignition power supply corresponds to the driving power supply, and if the vehicle is an electric vehicle, the system main relay corresponds to the driving power supply. The concept of an electric vehicle includes not only electric vehicles and battery-powered vehicles, but also vehicles equipped with both a motor and an engine as a drive source, such as hybrid vehicles and plug-in hybrid vehicles. A trip refers to the series of drives from when the driving power supply is turned on until it is turned off.

[0048] In this embodiment, the camera ECU1 is provided with a storage memory 15 as a destination for the concatenated image CI, but this is not limited to this configuration. The storage memory 15 may be provided by another ECU. Alternatively, the storage memory 15 may be provided by a server. In other words, the storage destination for the concatenated image CI may be a server / database located outside the vehicle. In that case, the storage processing unit F44, in cooperation with a communication ECU or the like, sends the concatenated image CI to an external server and performs processing to save it to a predetermined database.

[0049] The recognition processing unit F45 is configured to detect the position and type of a predetermined object by analyzing the image input from camera 2. The recognition processing unit F45 functions as a classifier that identifies the type of object based on, for example, the feature vector of the image. The recognition processing unit F45 performs object identification using, for example, CNN (Convolutional Neural Network) or DNN (Deep Neural Network) technology that applies deep learning. Objects to be detected include pedestrians and other vehicles, as well as road markings and road edges. Road markings include lane markings, stop lines, and arrow lines indicating the direction of travel at intersections. Objects to be detected may also include traffic signs such as directional signs and structures attached to the road such as traffic lights. If the recognition processing unit F45 recognizes a preceding vehicle, it can also determine the distance to the preceding vehicle. For the recognition of the preceding vehicle and the estimation of the distance between vehicles, the detection results of millimeter-wave radar, LiDAR, or sonar may be used. In this context, "leading vehicle" refers to the vehicle closest to your vehicle among other vehicles traveling in the lane ahead of your vehicle. The recognition result from the recognition processing unit F45 is output to, for example, the driver assistance ECU.

[0050] Figure 7 summarizes the operation of the data processing unit F4 in generating a concatenated image, and may include steps S11 to S15. Step S11 is the step in which the base image acquisition unit F41 sequentially acquires image frames generated based on the video signal as image data for concatenated image generation at predetermined sampling intervals. Step S12 is the step in which the region of interest extraction unit F42 extracts a partial image PI from the image frames acquired in step S11. Step S13 is the step in which the concatenation processing unit F43 performs the process of joining the partial images PI together.

[0051] Step S14 is a step in which it is determined whether or not the save execution conditions, which are the conditions for saving the concatenated image CI, have been met. Possible save execution conditions for concatenated image CI include the number of concatenations / concatenation length Lc being equal to or greater than a predetermined value, the driving power supply being turned off, or the detection of an instruction operation to terminate recording. Step S15 is a step in which the save processing unit F44 saves the concatenated image CI that has been generated at that time to the save memory 15 if the save execution conditions have been met.

[0052] <Specific Example (1)> This section explains, using Figures 8 and 9, an example of recording settings and the operation of the data processing unit F4 when the purpose is to check the time-dependent changes in weather conditions (sky conditions). Figure 8 shows image frames with different capture times extracted at a predetermined sampling interval, and Figure 9 shows an example of a concatenated image CI generated from these. Figure 9 is shown enlarged from its original size for better visibility. As mentioned above, since weather conditions do not change in units of a few seconds, the sampling interval can be set to 10 seconds. In Figures 8, etc., "t" indicates the elapsed time since the start of recording, and the unit is seconds.

[0053] In Figure 8, the dashed lines superimposed on each image frame indicate the cropping area CR and are not included in the actual image frame. The cropping area CR is set near the top edge so that weather conditions can be extracted. The cropping width Wc can be set to, for example, 180px. Note that Figure 8 shows the case where the cropping area CR is set a predetermined amount below the top edge of the image frame, but of course, the cropping area CR may also be set to align with the top edge of the image frame.

[0054] According to this embodiment, as shown in Figure 9, image data showing weather conditions at multiple points in time can be obtained as a concatenated image CI. This concatenated image CI allows for a quick understanding of the changes in weather conditions before and after the occurrence of important events detected separately. Important events include contact with other moving / stationary objects, as well as sudden steering and sudden braking. The vehicle may also be equipped with an automated driving system. The automated driving system is a device that autonomously drives the vehicle based on recognition results from surrounding monitoring sensors such as cameras and LiDAR. LiDAR stands for Light Detection and Ranging, or Laser Imaging Detection and Ranging. Important events related to automated driving include the termination of the automated driving mode, the execution of notification processing to transfer driving authority from the automated driving system to the driver, and the start of MRM (Minimal Risk Maneuver). The concatenated image CI generated by the camera ECU1 of this disclosure can be used as a record showing the conditions during automated driving.

[0055] Sudden steering corresponds to the yaw rate / steering speed exceeding a predetermined value. Similarly, sudden braking corresponds to the deceleration exceeding a predetermined value. MRM is a vehicle control system that is executed when the driver is unable to drive in a situation where the driver should be driving. Specifically, MRM can involve, for example, autonomously driving the vehicle to a safe location while issuing warnings to the surroundings, and then stopping it. Safe locations include road shoulders with a width of a predetermined value or more, zebra zones, and areas designated as emergency evacuation areas. The content of RM may also involve gradually decelerating and stopping the vehicle within the lane it is currently traveling in.

[0056] <Specific Example (2)> Next, using Figures 10 and 11, we will explain an example of the operation settings of the data processing unit F4 and its function when the purpose is to check the time-dependent changes in road surface conditions, such as whether or not the road surface is wet. Figure 10 shows image frames extracted at 10-second intervals with different capture times, and Figure 11 shows an example of a concatenated image CI generated from these. Like Figure 9, Figure 11 is shown enlarged from its original size for better visibility.

[0057] In Figure 10, the dashed lines superimposed on each image frame indicate the cropping area CR and are not included in the actual image frame. The cropping area CR is set below the center of the image frame so that the road surface condition as the object of interest can be recorded. The cropping width Wc can be set to, for example, 360px. The position of the cropping area CR in the vertical direction can be adjusted as appropriate according to the imaging direction of camera 2. The cropping area CR is set to correspond to the road surface range from a point a first distance ahead of the front end of the vehicle to a point a second distance ahead. For example, the first distance can be set to 8m and the second distance to 12m. With this setting, road surface images from 8m ahead to 12m ahead of the vehicle will be recorded. If the cropping area CR is too low, the body of the vehicle may be captured depending on the mounting position of camera 2, etc. Therefore, it is preferable that the cropping area CR is positioned a predetermined amount above the bottom edge of the image frame. Also, if the cropping area CR is too high, the road surface condition may be difficult to record depending on preceding vehicles, etc. Therefore, it is preferable that the CR excision area be positioned a predetermined amount below the center of the image frame.

[0058] According to this embodiment, as shown in Figure 11, image data is obtained that shows the road surface conditions at multiple points in time as a concatenated image CI. With this concatenated image CI, it becomes possible to grasp the changes in the road surface conditions over time within a predetermined time before and after the time of an event at a glance.

[0059] <Specific Example (3)> In the above specific examples (1) and (2), we illustrated how to extract a partial image PI in the horizontal direction. However, as mentioned above, the region of interest extraction unit F42 may also be set to extract a partial image PI in the vertical direction. Here, we will explain the operation of the region of interest extraction unit F42 when it extracts an image in the vertical direction using Figures 12 and 13. Figure 12 shows image frames with different capture times, extracted at 10-second intervals, similar to Figures 8 and 10. Figure 13 shows an example of a concatenated image CI formed by horizontally concatenating these partial images PI. Like Figure 9, Figure 13 is shown enlarged from its original size for better visibility.

[0060] In Figure 12, the dashed lines superimposed on each image frame indicate the cropping area CR and are not included in the actual image frame. The cropping area CR is set to pass through the center of the image frame. The cropping width Wc is set to, for example, 340px. Of course, the cropping width Wc may also be 170px, etc. The cropping area CR is set to divide the image frame into left and right halves. Note that Figure 10 shows the case where the cropping area CR is set to pass through the center of the image frame, but of course, the cropping area CR may also be set at a predetermined distance to the left or right of the center of the image frame.

[0061] With these recording settings, as shown in Figure 11, image data is obtained that collectively shows road surface conditions and weather conditions at multiple points in time as a concatenated image CI. This concatenated image CI makes it possible to grasp the changes in road surface conditions and weather conditions over time within a predetermined period before and after the time of an event at a glance.

[0062] Furthermore, by configuring the cropping area CR to correspond to a predetermined row range in the direction in front of the vehicle, it becomes possible to record the presence or absence of a preceding vehicle and the distance between them. With the above settings, it becomes possible to grasp at a glance the changes in the conditions in front of the vehicle, such as the presence or absence of a preceding vehicle, within a predetermined time before and after the event occurrence. Based on the above effects, if the preceding vehicle is the object of interest, it is preferable to set the cropping direction to the vertical direction and the horizontal position of the cropping area CR to include the center of the frame.

[0063] Incidentally, the presence or absence of a preceding vehicle, the distance between vehicles, and the vehicle that constitutes a preceding vehicle can change dynamically compared to weather conditions. Therefore, when the object of interest is a preceding vehicle, it is preferable to set the sampling interval to seconds, such as 1 second or 2 seconds. Of course, when the object of interest is a preceding vehicle, the sampling interval may also be less than 1 second, such as 250 milliseconds or 500 milliseconds. The extraction width Wc when the object of interest is a preceding vehicle should also be set appropriately so that the desired information can be recorded.

[0064] With the configuration described above, only a portion of the image frames generated by camera 2 is saved, thus reducing the data size stored in storage memory 15. Consequently, the requirements for storage memory 15 can be relaxed. Furthermore, since the number of times data is saved to storage memory 15 can be reduced, the usable period (so-called lifespan) of the memory can be extended. In addition, since the situation at each time point can be viewed at a glance in a single image, there is no need to spend time reviewing recorded video data. In other words, it is possible to improve the efficiency of the work of checking the changes in the situation of the subject of interest over time. Moreover, the camera ECU 1 described above is not an event-recording type, but rather a type of so-called continuous recording recorder that records images at predetermined sampling intervals. Therefore, it is possible to reduce the risk of the subject of interest not being recorded compared to an event-recording type recorder.

[0065] While embodiments of the present disclosure have been described above, the present disclosure is not limited to the embodiments described above. Various modifications described below are also included within the technical scope of the present disclosure, and further modifications can be made in various ways without departing from the gist of the invention. For example, the various supplements and modifications described below can be combined as appropriate without causing any technical inconsistencies. In addition, components having the same function as those described above may be denoted by the same reference numerals, and their descriptions may be omitted. Also, if only a part of the configuration is referred to, the above description may be applied to the other parts.

[0066] <Example (1)> The above describes the case where camera 2 is a camera that captures images in front of the vehicle, but it is not limited to this. Camera 2 may also be a camera that captures images behind the vehicle, or a camera that captures images to the side of the vehicle.

[0067] Furthermore, a driver monitoring camera positioned to capture the driver's face may also be used. Here, "driver" primarily refers to the person seated in the driver's seat of the vehicle. However, a person seated in a remote control seat outside the vehicle (a so-called operator) can also be included in the concept of a driver. Driver monitoring cameras are positioned, for example, on the top surface of the steering column cover, the top surface of the instrument panel, the rearview mirror, or the upper edge of the windshield.

[0068] If camera 2 is a driver monitoring camera, the driver's eyes can be set as the object of interest. The position of the cropping region CR in the vertical direction is adjusted manually or automatically to include the position of the driver's eyes. For example, when the recognition processing unit F45 detects the driver's eyes, the height position of the cropping region CR can be dynamically adjusted to correspond to the detected position of the driver's eyes. By extracting the image region including the driver's eyes as a partial image PI and saving it as a concatenated image CI, it is possible to record the change in the driver's alertness over time. Accordingly, the change in the driver's degree of alertness before and after a major event such as an accident can also be easily verified. When the driver's eye opening is the object of recording (object of interest), the sampling interval can be set to 1 second, 2 seconds, 4 seconds, etc.

[0069] <Modification (2)> Camera ECU1 may generate a concatenated image CI based on an image in which a marker indicating the recognition result of the recognition processing unit F45 is superimposed on the image captured by camera 2. To accommodate this technical concept, camera ECU1 may also include an image processing unit F46 as shown in Figure 14. The image processing unit F46 is configured to generate a composite image by adding recognition results, timestamps, etc., to the image from camera 2. The image processing unit F46 may also have a function to apply distortion correction, keystone correction, etc., to the image captured by camera 2 according to the lens characteristics. The base image acquisition unit F41 can acquire the processed image generated by the image processing unit F46 as a base image for generating a concatenated image. The image processing unit F46 may also be called an image synthesis unit or a video processing unit.

[0070] Such an image processing unit F46 may be integrated with the base image acquisition unit F41 as a sub-function unit of the base image acquisition unit F41. Acquisition here includes generation / detection by internal calculations. The image processing unit F46 may also be provided by the video acquisition unit F1. Furthermore, the image processing unit F46 may be interposed between the video acquisition unit F1 and the base image acquisition unit F41.

[0071] The image processing unit F46 generates an image by superimposing other vehicle markers on the image frame input from camera 2, for example, to highlight other vehicles. The other vehicle markers are rectangular frames that surround other vehicles recognized by the recognition processing unit F45. The other vehicles to which the other vehicle markers are added may be only preceding vehicles or only oncoming vehicles. The image processing unit F46 may also be configured to add other vehicle markers only to other vehicles whose collision risk, calculated separately, is above a predetermined value. Other vehicles with a collision risk above a predetermined value refer to vehicles whose remaining collision time (so-called TTC: Time-To-Collision) or collision margin (MTC: Margin-To-Collision) is below a predetermined value.

[0072] The recognition processing unit F45 can recognize not only other vehicles but also pedestrians and stationary objects. The image processing unit F46 may generate images with markers added to pedestrians and predetermined stationary objects. The image processing unit F46 may also generate images with lane recognition lines superimposed on image frames input from camera 2. The lane recognition lines are linear markers that indicate the position of lane markings recognized by the recognition processing unit F45.

[0073] Figure 15 conceptually illustrates the operation of the camera ECU1 in this modified example. The base image acquisition unit F41 acquires images with a marker Mk indicating the recognition result of a preceding vehicle, etc., at predetermined sampling intervals, as shown in Figure 15(A). The region of interest extraction unit F42 in this modified example extracts a predetermined region from a composite image frame that may contain a marker, as shown in Figure 15(B). Of course, the presence or absence of a marker is affected by whether or not a recognition target exists, so the image frame from which the partial image PI is extracted may not contain a marker. The concatenation processing unit F43 generates a concatenated image CI from multiple partial images PI, as shown in Figure 15(C), similar to the embodiment described above. With this configuration, the recognition status of the recognition processing unit F45 for a predetermined object can be recorded.

[0074] Figures 16 and 17 illustrate the operation of the camera ECU1 when the recognition status of lane markings at a predetermined verification distance ahead of the vehicle is set as the object of interest. The term "point" here includes the concept of a section or area having a predetermined length. Figure 16 shows image frames at multiple points in time with marker Mk superimposed as lane recognition lines, and Figure 17 shows an example of a concatenated image CI formed by connecting these partial images PI laterally.

[0075] The verification distance can be a relatively short distance such as 8m, 10m, or 16m, or a relatively long distance such as 25m, 30m, or 40m. With the above configuration, the progress of the recognition results that are of interest to the developer / verifier can be grasped at a glance. In addition, it becomes easy to find locations where the recognition of the road markings fails, even though the road markings are visible. The sampling interval can be 10 seconds, or it can be a few seconds such as 1 or 2 seconds. Furthermore, the sampling interval can be 250 milliseconds or 500 milliseconds. The cropping width Wc can be 180px or 360px, etc.

[0076] <Examples of application of modified form (2)> The region of interest extraction unit F42 may dynamically change the extraction width Wc according to the recognition result of the object set as the object of interest. For example, if a preceding vehicle is set as both the object of recognition and the object of interest, the region of interest extraction unit F42 may dynamically set the extraction width Wc to include the preceding vehicle if the preceding vehicle is recognized. Figure 18 is a conceptual diagram showing the operation of the data processing unit F4 when the extraction width Wc is dynamically adjusted to include the preceding vehicle.

[0077] This configuration allows developers to grasp the temporal changes of a recognition target of their interest (e.g., a preceding vehicle) at a glance. It also enables efficient verification of the accuracy of the recognition logic. For example, it makes it easier to identify scenes where the preceding vehicle is likely to be lost. Even in configurations where the cropping width Wc is dynamically adjusted, the basic cropping width Wc and cropping region CR are set. The region of interest cropping unit F42 adjusts the cropping width Wc so that it includes a predetermined basic setting range of the cropping region CR. The basic setting range refers to the row range / column range that should be cropped when no recognition target exists. In a configuration that crops a partial image PI horizontally, adjusting the cropping region CR to include the recognition result is equivalent to adjusting the cropping start position higher and the cropping end position lower, as needed. The cropping start position refers to the uppermost row number of the cropping region CR, and the cropping end position refers to the lowermost row number of the cropping region CR.

[0078] <Variation (3)> The camera image processing system Sys may include multiple cameras 2 that capture images of the area outside the vehicle. For example, as shown in Figure 19, the camera ECU 1 is connected to four cameras 2: a front camera 2A, a rear camera 2B, a left camera 2C, and a right camera 2D. These four cameras 2 are positioned at different locations within the vehicle and capture images of different directions around the vehicle. Specifically, they are as follows:

[0079] The front camera 2A is a camera that captures images of the area in front of the vehicle at a predetermined angle of view. The front camera 2A is mounted at the front end of the vehicle, such as the front grille, with its optical axis facing forward. The rear camera 2B is a camera that captures images of the area behind the vehicle at a predetermined angle of view. The rear camera 2B is positioned at a predetermined location on the rear of the vehicle, such as near the rear license plate or rear window, with its optical axis facing backward. The left camera 2C is a camera that captures images of the left side of the vehicle. The left camera 2C is mounted at the left side mirror or left pillar, for example, with its optical axis facing left of the vehicle. The right camera 2D is a camera that captures images of the right side of the vehicle. The right camera 2D is mounted at the right side mirror or right pillar, for example, with its optical axis facing right of the vehicle. Of course, the mounting position and orientation of each camera 2 are just examples and can be changed as appropriate.

[0080] These cameras 2 employ wide-angle lenses such as fisheye lenses, and each camera 2 has a field of view of 180 degrees or more. Therefore, by using the four cameras 2, it is possible to capture the entire surroundings of the vehicle. The camera image processing system Sys may also include a camera 2 mounted on the roof. The mounting positions of the left camera 2C and right camera 2D mentioned above are not limited to the side mirrors, but may also be on the rooftop, etc. Some or all of the multiple cameras 2 may be retrofitted cameras, for example, on the roof, on the dashboard, near the window frame, etc. Each of the multiple cameras 2 outputs a video signal based on the captured image to the camera ECU 1. The data processing unit F4 of the camera ECU 1 may individually generate the above-described concatenated image CI for each camera 2. Since the imaging range differs for each camera 2, parameters such as the cropping direction and cropping width Wc may differ for each camera 2.

[0081] <Example of application of modified form (3) (1)> The data processing unit F4 may be configured to extract partial images PI from an integrated image of images captured by multiple cameras 2 and generate a concatenated image CI. As a prerequisite, the data processing unit F4 includes an image processing unit F46. Figure 20 shows an overview of the operation of the camera ECU1 in this application example. As shown in Figure 20(A), the image processing unit F46 extracts images captured by each of the multiple cameras 2 at a common sampling interval based on the video signals provided by each of the cameras 2. The image frames extracted from each camera 2 are images captured at approximately the same time.

[0082] Next, as shown in Figure 20(B), the image processing unit F46 combines the image frames captured by each camera 2 at the same time in a predetermined order. Figure 20 shows a configuration in which image frames are combined horizontally. In another configuration, the image processing unit F46 may combine the image frames vertically. For convenience, an image frame formed by connecting image frames captured by multiple cameras 2 is referred to as a concatenated frame. The direction in which the image frames are combined is the same as the cropping direction. The concatenated frames generated by the image processing unit F46 are provided to the base image acquisition unit F41. That is, the base image acquisition unit F41 acquires the concatenated frames from the image processing unit F46 at a predetermined sampling interval.

[0083] Then, as shown in Figure 20(B), the region of interest extraction unit F42 generates a partial image PI, which is the basis for the concatenated image CI, by extracting the portion set as a predetermined extraction region CR in the concatenated frame. The partial image PI corresponds to a portion of the image captured by each camera 2.

[0084] The concatenation processing unit F43 adds the newly generated partial image PI to the end of the existing concatenated image CI each time the region of interest extraction unit F42 generates a partial image PI. The concatenation processing unit F43 may also be configured to concatenate multiple partial images PI in chronological order at predetermined timings.

[0085] As described above, the image acquired by the base image acquisition unit F41 may be a combination of images acquired from multiple cameras. With the above configuration, the temporal changes in image data captured by multiple cameras can be grasped at a glance. The subjects of interest for this modified example / application are expected to be road surface conditions, distance to surrounding vehicles, and recognition status of pedestrians, etc.

[0086] Incidentally, Figure 20 describes an embodiment in which image frames from each of the two cameras are concatenated and then partial image PIs are extracted, but the processing procedure is not limited to this. As shown in Figure 21, the region of interest extraction unit F42 may extract partial image PIs of the same size from each camera image, and the concatenation processing unit F43 may connect them in a predetermined order in the extraction direction to generate the final partial image PI, which is the camera integrated partial image CPI. The concatenation processing unit F43 generates a concatenated image CI by connecting the sequentially generated camera integrated partial image CPIs in chronological order in the concatenation direction.

[0087] Note that the cropping area CR may differ for each camera 2, as shown by the dashed line in Figure 21(B). This is because the area in which the object of interest is captured may differ for each camera 2. However, in order to combine the images, the cropping width Wc and cropping direction should be the same value / direction. According to this modified example / application, it becomes possible to save the image area that may contain the object of interest more efficiently.

[0088] <Application example (2) of modified example (3)> The image processing unit F46 may have a function to generate a bird's-eye view image based on images from multiple cameras 2, as shown in Figure 22. The bird's-eye view image is an image of the area around the vehicle viewed from a virtual viewpoint set in the sky. The partial images PI that constitute the concatenated image CI may be predetermined regions of the bird's-eye view image. In other words, the camera ECU 1 may generate a concatenated image CI by extracting predetermined regions of the bird's-eye view image generated from multiple camera images at predetermined sampling intervals and sequentially connecting them. In such a configuration, the base image acquisition unit F41 acquires the bird's-eye view image generated by the image processing unit F46 at predetermined sampling intervals. The multiple bird's-eye view images acquired sequentially by the base image acquisition unit F41 have different generation times at the sampling intervals. The region of interest extraction unit F42 extracts a predetermined extraction region CR as a partial image PI from the bird's-eye view image acquired by the base image acquisition unit F41. The concatenation processing unit F43 sequentially generates and updates the concatenated image CI by connecting the partial images PI generated sequentially by the region of interest extraction unit F42 in chronological order in the concatenation direction. This configuration, which generates concatenated image CIs based on bird's-eye view images, can improve the recognition of changes in road surface conditions over time.

[0089] Furthermore, in the bird's-eye view image, if the image range corresponding to the cropping region CR is in front of the vehicle and does not include image data from the rear camera 2B, the image processing unit F46 may generate a bird's-eye view image specialized for the front of the vehicle without using the image frame from the rear camera 2B. Also, in the bird's-eye view image, if the image range corresponding to the cropping region CR can be generated using only the front camera 2A, the image processing unit F46 may generate an image obtained by transforming the viewpoint of the image from the front camera 2A. The viewpoint transformation process is a process that transforms a captured image into an image viewed from a virtual viewpoint by mapping it to a predetermined projection plane, etc. The base image acquisition unit F41 may be configured to acquire from the image processing unit F46 the image data obtained by the image processing unit F46 after the viewpoint transformation process has been applied to the image captured by camera 2, as base image data.

[0090] Even in configurations where partial image PIs are extracted from image frames that have undergone viewpoint transformation, such as bird's-eye views, and saved as concatenated images, the technical concept of saving the images with markers indicating the recognition results superimposed is applicable. For example, the base image acquisition unit F41 may acquire an image frame as the base image frame in which markers indicating the recognition results are superimposed on a processed image that has undergone viewpoint transformation, and the region of interest extraction unit F42 may extract partial image PIs from this image frame. The extraction width Wc may also be dynamically changed according to the recognition results.

[0091] A processed image is an image that has been subjected to predetermined processing, such as viewpoint transformation, distortion correction, superposition of text / markers / timestamps, brightness correction, and color tone adjustment, on an image captured by one camera 2. In one embodiment, it is also possible to interpret processed images more restrictively to images that have undergone viewpoint transformation and images to which supplementary information such as markers has been added. Furthermore, a composite image is an image that combines one image with another image. A typical example of a composite image is a bird's-eye view image. Composite images include not only images that combine multiple images with different output sources, but also images in which pre-prepared image elements such as markers have been superimposed on a single image. In other words, a composite image can be understood as a type of processed image. Underbody transparency images are also a type of composite image. An underbody transparency image is an image that shows the underside of the vehicle body by making it transparent, etc., by combining past images from the forward camera 2A acquired when the vehicle is moving forward with current images from other cameras. The virtual viewpoint of composite images such as underbody transparency images is not limited to the air above the vehicle, but may also be the driver's viewpoint, in other words, the driver's eye box.

[0092] <Example (4)> The camera ECU1 may be configured to change the values ​​of data recording settings from default values ​​to temporarily applied values ​​in certain scenes. Data recording settings refer to the sampling period, crop width Wc, etc. Default values ​​refer to the settings applied in normal scenes. Temporarily applied values ​​are settings applied when a scene is determined to be a concentrated recording scene. Normal scenes refer to scenes other than concentrated recording scenes. Concentrated recording scenes are scenes where data is stored more densely than in normal scenes. Normal scenes are scenes where the data storage density is less dense than in concentrated recording scenes.

[0093] Examples of centralized recording scenes in which the recording method is changed include (i) when passing through an intersection, (ii) when driving at low speed, (iii) when parking, and (iv) when exiting a parking space. (i) When passing through an intersection can be further subdivided into three categories: (a) when turning right, (b) when turning left, and (c) when driving straight. Driving at low speed refers to a state in which the speed of movement is below a predetermined low-speed threshold. The low-speed threshold is, for example, 10 km / h or 20 km / h. Driving at low speed can also include autonomous driving for parking. (iii) When parking and (iv) when exiting a parking space can also be understood as examples of (ii) when driving at low speed. In this disclosure, a state in which the speed of movement is above the low-speed threshold is also referred to as the normal driving state. The normal driving state includes driving at high speed. Driving at high speed refers to a state in which the speed exceeds a predetermined high-speed threshold, such as 60 km / h or 80 km / h.

[0094] The temporary applied value only needs to be different from the default setting. The temporary applied value can be set to a value that allows for more information to be recorded than in a normal scene. For example, if the default setting for the sampling interval is 10 seconds, the temporary applied value can be set to 2 seconds, 1 second, or 500 milliseconds. The temporary applied value for the sampling interval can be set to less than half of the default setting. Also, if the default setting for the crop width Wc is 180px, the temporary applied value can be set to 360px, etc. The temporary applied value for the crop width Wc can be set to 1.5 times or more, more preferably 2 times or more, the default setting.

[0095] Figure 23 is a flowchart illustrating an example of the operation of the camera ECU1 corresponding to this modified example. The processing flow shown in Figure 23 is started, for example, when it is detected that the vehicle's power supply is turned on and that the user or other person has instructed the start of data recording. Of course, in settings where recording is always performed when the power supply is on, the flow shown in Figure 23 may be started based on the power supply being turned on. Below, as an example, we will describe the case where passing through an intersection is set as the concentrated recording scene. Note that, in parallel with / independent of this flow, the processor 11 extracts partial images PI and updates the concatenated images CI at the set sampling interval.

[0096] First, the processor 11 determines whether or not the scene corresponds to a centralized recording scene based on information input from the on-board sensors 4 and the on-board ECU 5, such as vehicle speed and the operating status of the turn signals. More specifically, the processor 11 determines whether or not the vehicle is passing through an intersection based on input signals from the on-board sensors 4, etc. Image recognition results for features such as road markings can also be used as a basis for determining the scene. In addition, the processor 11 may acquire information about the planned route and the road traveled from the navigation system or the driver assistance ECU and determine the scene based on that information.

[0097] If the determination process in step S21 determines that the vehicle is passing through an intersection, step S23 is executed. On the other hand, if it is determined that the vehicle is not passing through an intersection, step S22 is executed. Step S22 is a step to maintain the default settings. Step S23 is a step to change various settings related to data recording from default settings to temporarily applied values.

[0098] While the temporary values ​​are in use, the processor 11, in step S24, sequentially determines whether the concentrated recording scene is continuing based on signals from the on-board sensors 4, etc. More specifically, the processor 11 determines whether the vehicle is still passing through an intersection based on the vehicle speed, yaw rate, vehicle position information, and road marking recognition status.

[0099] When a concentrated recording scene ends, for example, when it is detected that the vehicle has exited an intersection, step S25 resets various settings related to data recording from temporary applied values ​​back to default settings. Step S26 is a step in which it is determined whether or not a recording termination operation has been performed, based on a signal from the input device 3. A recording termination operation refers to actions such as pressing a switch to stop / terminate data recording. Note that an operation to turn off the driving power supply can also be included as a recording termination operation. The determination of whether or not a recording termination operation has been performed can also be performed while a concentrated recording scene is in progress.

[0100] When the recording termination operation is detected, the processor 11 confirms the concatenated image CI and sends it to a predetermined server or the like (step S26). If a tablet terminal or notebook computer is connected to the vehicle / camera ECU 1 via wired / wireless connection, the camera ECU 1 may also send concatenated image data to the connected terminal as a trigger when the recording termination operation is detected.

[0101] According to the above configuration, in specific scenes, information about the object of interest can be recorded more densely than in normal scenes. If the object of interest is the recognition state of other moving objects when passing through an intersection, data during intersection passage can be recorded at a sufficiently long interval, and the amount of data in scenes other than intersection passage can be further reduced. As a result, it becomes possible to improve the efficiency of analysis of the object of interest while suppressing the amount of data to be stored. The above configuration corresponds to a configuration in which the data storage density is changed between specific scenes and others. Furthermore, the above configuration corresponds to a configuration in which at least one of the sampling interval and the crop width Wc is dynamically changed according to the driving scene / vehicle speed. Configurations that change the sampling interval may also include a configuration in which recording is effectively stopped by setting the sampling interval to a sufficiently large value equivalent to infinity.

[0102] Furthermore, the camera ECU1 may be configured to generate and update concatenated image CI at a predetermined sampling interval only when it is a specific scene designated as a recording target scene. In other words, even when driving, the camera ECU1 may be configured to stop extracting partial image PI and updating concatenated image CI if the scene does not fall under the recording target scene. Scenes such as those exemplified as concentrated recording scenes can be used as recording target scenes. With this configuration, since data is not collected for scenes other than the recording target scenes, the amount of data stored can be further reduced. Note that setting a recording target scene is, paradoxically, equivalent to setting scenes that will not be recorded, or scenes that will not be recorded. The above configuration is equivalent to a configuration in a camera ECU1 that is configured to allow registration of scenes that will not be recorded, where data recording is stopped when a scene that will not be recorded falls under the category of a scene that will not be recorded.

[0103] <Variation (5)> The camera ECU1 may have multiple operating modes. For example, the camera ECU1 may be configured to allow switching between a non-recording mode, a first recording mode, and a second recording mode. The non-recording mode is an operating mode in which no image is saved. The first recording mode is an operating mode in which the original image frame is saved as is, rather than partial images. The first recording mode may also be a mode in which video data is saved. The second recording mode is an operating mode in which multiple partial images are saved as a concatenated image. The camera ECU1 operating in the second recording mode may correspond to the various configurations described above as embodiments / modifications.

[0104] In this modified example, the input device 3 may include a switch for switching operating modes. For example, the input device 3 may include a toggle switch for switching operating modes. This switch may be a mechanical switch or an image button realized through the cooperation of a touch panel and a display. The operation reception unit F2 receives user instructions to change the operating mode based on signals from the input device 3.

[0105] The camera ECU1 may automatically change its operating mode depending on the available space in the storage memory 15. For example, if the available space in the storage memory 15 falls below a predetermined value, the camera ECU1 may automatically change its operating mode from the first recording mode to the second recording mode. When the operating mode is automatically switched, it is preferable for the camera ECU1 to notify the user of the change in operating mode by display or sound, as described below.

[0106] The camera ECU1 may be connected to a display device 6 and a speaker 7, as shown in Figure 24, as a device for notifying the user of the operating mode. The display device 6 displays an icon indicating the operating mode of the camera ECU1 based on the input signal from the camera ECU1. The display device 6 may be a meter display or a head-up display. A meter display is a display provided in the area of ​​the instrument panel located in front of the driver's seat, and can be realized using a liquid crystal panel or an organic light-emitting (electroluminescence: EL) panel. A head-up display is a device that projects light representing an image (hereinafter referred to as image light) onto a projection member such as the vehicle's windshield, thereby displaying a virtual image at a position in front of the vehicle as seen by the driver. The icon indicating the operating mode may be displayed at all times while the driving power is on. The icon indicating the operating mode may be displayed only for a certain period of time from the moment the operating mode changes.

[0107] The status of the camera ECU1 may be notified to the driver not only by an icon display, but also by a text display. Furthermore, the status of the camera ECU1 may be represented by the illumination of an LED (color and whether it blinks). For example, the camera ECU1 may turn off its LED in non-recording mode, illuminate its LED in first recording mode, and blink its LED in second recording mode. The status of the camera ECU1 may also be notified to the driver by an audio message output from speaker 7. When the operating mode of the camera ECU1 switches, it may output an announcement / notification sound from speaker 7 to inform the driver of the changed operating mode.

[0108] <Differentiation (6)> As shown in Figure 25, the camera ECU1 may also include an inspection processing unit F47 that detects abnormalities in a predetermined object to be inspected based on a partial image. The camera ECU1, acting as the inspection processing unit F47, detects abnormalities in the object to be inspected in a partial image by comparing the most recent image taken at the same location with past images.

[0109] In this disclosure, "latest image" refers to a partial image newly generated during the current trip. "Latest image" can be rephrased as "current image." "Past image" in this disclosure refers to a partial image generated and saved during a previous trip. Generally speaking, a past image refers to a partial image generated one day or more ago.

[0110] In this disclosure, "objects to be inspected" refers to equipment / land features that should be inspected by image comparison to determine whether or not there are any abnormalities such as damage or deterioration. Objects to be inspected correspond to specific examples of the object of interest. Objects to be inspected may be a variety of equipment installed along the road, such as road signs, road markings, traffic lights, electronic display boards, utility poles, power lines, and manholes. Road signs may be classified into lateral signs and overhead signs depending on their orientation as viewed from camera 2. Lateral signs refer to signs placed on the right or left side of the road in a space where the height from the road surface is less than 3m. Lateral signs may include so-called roadside signs, which are signs with a signboard mounted on a pole installed on the roadside, in the middle of the road, on a sidewalk, or in a median strip. Lateral signs may include some or all of regulatory signs, warning signs, directional signs, and auxiliary signs. Overhead signs refer to signs of the type in which the signboard is placed 4.7m or more above the road surface using a column or gate-type support. Overhead signs can include not only gantry-type signs but also cantilevered and mounted signs. A specific example of an overhead sign is a guide sign. Some guide signs may be classified as side signs.

[0111] When the upper marker is set on the object to be inspected, the extraction direction is horizontal, and the extraction region CR is set near the upper end. That is, when the upper marker is set on the object to be inspected, the camera ECU1 extracts the area near the upper end of the original image frame as a partial image, as shown in Figure 26, and saves it concatenated. The extraction width can be set to any value. Note that concatenated saving here refers to the process of saving partial images generated at other times in chronological order, as described above.

[0112] Each partial image is stored in the storage memory 15, associated with shooting condition data. Here, shooting condition data refers to a dataset of items indicating the conditions at the time of partial image acquisition (i.e., shooting conditions). For example, as shown in Figure 27, the shooting condition data includes data such as the partial image number, shooting time, shooting location, direction of travel, weather, and preceding vehicle. The partial image number is an identifier indicating which partial image the data pertains to, either from the partial images in the concatenated image or from the partial images generated during the current trip. Note that the shooting condition data for each partial image may also be stored linked to the partial image data itself; in this case, the partial image number may be omitted.

[0113] The shooting location is the positional coordinate of the shooting point, and is determined based on the output signal of the GNSS receiver. The shooting location may be expressed in the World Geodetic System (WGS84) or in other coordinate systems. The shooting location may be expressed in two-dimensional coordinates of latitude and longitude, or it may include altitude information. A configuration that includes altitude information can suppress misidentification of the shooting location in elevated roads and road sections with double-deck structures.

[0114] The direction of travel can be expressed as an azimuth angle. The direction of travel may also be expressed as the yaw angle of the vehicle body with respect to the direction of extension of the lane markings that define the lane the vehicle is traveling in. This direction of travel may be detected by an azimuth sensor or identified by image recognition. Weather information may be determined from the operation status of a rain sensor or wipers, or identified based on weather information received from an external device. Weather can be expressed by a code. A code of 00 can be used to indicate sunny weather, a code of 01 to indicate rain, etc., as appropriate.

[0115] Furthermore, the shooting status data may include information on the preceding vehicle, indicating whether or not there was anything obstructing the view of camera 2 in front of the vehicle. The preceding vehicle information indicates whether or not a vehicle equivalent to the preceding vehicle was present within a certain distance from the vehicle. The presence or absence of the preceding vehicle can be represented by a flag (0 / 1) or a code. The preceding vehicle information may also include the distance between the vehicle and the preceding vehicle. The preceding vehicle information may also include information on the type or size of the preceding vehicle, such as whether or not the preceding vehicle was a large vehicle. The preceding vehicle information corresponds to information indicating the presence or absence of obstacles for camera 2 / image recognition, in other words, information indicating the possibility that the object to be inspected is obstructed by the preceding vehicle.

[0116] The preceding vehicle information may be represented by a simpler flag. The camera ECU1 may be configured to turn on the preceding vehicle flag in the shooting situation data only when a large vehicle is present within a certain distance (e.g., 10m) from its own vehicle. Such a preceding vehicle flag indicates the quality of the view from camera 2 and can also be called a view flag or occlusion flag.

[0117] Of course, the shooting conditions data does not need to include all of the above items, and some items may be omitted. The shooting conditions data may also include the detected value from the illuminance sensor, i.e., the external illuminance. The shooting conditions data may be embedded in the partial image as metadata for the partial image. The shooting conditions data may also be embedded in the partial image as text, as shown in Figure 6. The shooting conditions data may be saved in a separate file from the partial image. In that case, it is preferable that the partial image number, shooting time, and part of the shooting position coordinates are embedded in each partial image to facilitate correspondence with the shooting conditions data.

[0118] The camera ECU1, acting as the inspection processing unit F47, detects damage to the sign, which is the object being inspected, by comparing partial images taken at the same location, as shown in Figure 28. Figure 28 illustrates a case where a part of the upper sign has been deformed. In Figure 28, (A) shows a past image, and (B) shows the latest image. Note that the same location here is not limited to being exactly the same; an error of about 0.5m or 1.0m is acceptable.

[0119] Figure 29 is a flowchart illustrating an example of the operation of the camera ECU1, and may include steps S31 to S38. The processing flow shown in Figure 29 is performed, for example, at the sampling interval when the vehicle's power supply is on and set to the second recording mode.

[0120] Step S31 is the step in which the base image acquisition unit F41 acquires an image frame for partial image extraction based on the video signal. Step S32 is the step in which the region of interest extraction unit F42 extracts a partial image from the image frame acquired in step S31. Step S33 is the step in which the camera ECU1 concatenates and saves the partially image acquired this time with partially images generated at other times. At this time, shooting status data is linked to the partially image and saved in the storage memory 15.

[0121] Step S33 is a step in which it is determined whether or not the object to be inspected (in this case, the upper marker) is included in the partial image generated in step S32. This step is performed by the inspection processing unit F47 in cooperation with the recognition processing unit F45. The partial image generated in step S32 corresponds to the latest image.

[0122] Step S34 corresponds to the step of determining whether the latest image contains the object to be inspected. If the camera ECU1 cannot detect the object to be inspected from the latest image (S34 NO), this flow is terminated. On the other hand, if the camera ECU1 can detect the object to be inspected from the latest image (S34 YES), the subsequent processing (step S35 onwards) is executed.

[0123] Step S35 is a step in which the storage memory 15 is referenced and a partial image previously taken at the current location is read as a past image. If no past images are found as a result of the search for past images, such as when the vehicle is passing through the current location for the first time, this flow should be terminated.

[0124] Step S36 is a step in which the latest image is compared with a past image to determine whether or not there is a difference in the shape of the object being inspected. The image comparison may detect only differences in shape, or it may detect differences in color. If the configuration detects differences in color, it may be possible to detect the occurrence / spread of rust or graffiti. Difference detection based on image comparison is equivalent to a process that uses a past image as a reference (sample) to determine whether or not there is an abnormality in the object being inspected. For this reason, this process can also be called an image diagnostic process.

[0125] If the camera ECU1 detects a difference by comparing the latest image with past images (step S37 YES), it performs an abnormality notification process (step S38). The abnormality notification process may be a process that notifies the driver of the abnormality of the object being inspected using the display device 6 or speaker 7, or a process that wirelessly transmits a signal indicating that an abnormality has been detected in the object being inspected to a specific server. Abnormalities in the object being inspected may include not only damage, but also the accumulation of dirt, tilting above a predetermined value, etc.

[0126] Furthermore, if the preceding vehicle flag is turned on in the latest or past image, the object being inspected may not be captured in its original form due to occlusion. For this reason, the camera ECU1 may omit image diagnostic processing if the preceding vehicle flag is turned on in either the latest or past image. This configuration reduces the risk of mistakenly determining that there is an abnormality in the object being inspected when the latest / past image is missing part of the object being inspected due to occlusion. On the other hand, a configuration that performs image diagnostics on all latest images in which the object being inspected is captured, regardless of the presence or absence of a preceding vehicle, has the advantage of reducing the risk of overlooking potentially faulty equipment. The response policy of the camera ECU1 when the preceding vehicle flag is set to on may be configured to be changeable by the driver.

[0127] In addition, if the camera ECU1 detects an abnormality in the equipment as a result of image diagnostic processing using the latest or past images in which the preceding vehicle flag is set to ON, it may notify the driver / server that the diagnostic result is uncertain, along with the diagnostic result itself.

[0128] The above describes a method of concatenating and saving partial images regardless of whether the object to be inspected is captured in the cutout area CR, but it is not limited to this. The camera ECU1 may be configured to concatenate and save only when the object to be inspected is captured in the cutout area CR. For example, the execution order of steps S33 and S34 may be reversed. This configuration makes it possible to prevent the saving of partial images that are not related to the inspection of the equipment.

[0129] As mentioned above, the objects to be inspected are not limited to overhead signs, but may also include side signs or utility poles. When side signs or utility poles are set as the objects to be inspected, the cropping direction may be set vertically, and the cropping area CR may be set near the left edge or right edge of the image frame. When road markings such as stop lines are set as the objects to be inspected, the cropping direction may be set horizontally, and the cropping area may be set to the lower half, etc. The cropping direction and cropping area should be set to include the part of the image frame in which the objects to be inspected are expected to be captured.

[0130] <Example (7)> In the above modification (6), an embodiment was described in which image diagnosis is performed when it is detected that the latest image contains an upward sign, but the conditions for performing image diagnosis are not limited to this. As shown in Figure 30, the camera ECU 1 may be configured to acquire, save, and perform image diagnosis processing on a partial image as the latest image when the vehicle reaches a pre-registered inspection point. The inspection point is a point where the object to be inspected should be visible in the cropping area CR of the camera 2.

[0131] For convenience, the storage medium on which the list of inspection points is registered will be referred to as the inspection point storage unit. The camera ECU1 in this modified example is equipped with an inspection point storage unit and performs a series of processes related to image diagnosis based on the data stored in the inspection point storage unit. The inspection point storage unit may be implemented using a part of the storage area provided by the storage 13, or it may be a storage device independent of the storage 13. The inspection points may be registered in the storage 13 of the camera ECU1 as part of the program.

[0132] The inspection point is determined according to the installation location of the object to be inspected. The inspection point may be a point a predetermined distance before the object to be inspected. Therefore, the camera ECU1 may have data indicating the equipment installation location instead of data directly indicating the position coordinates of the inspection point. In that case, the camera ECU1 may calculate the inspection point based on the equipment installation location and the field of view of camera 2, and perform the flow shown in Figure 30. The data indicating the equipment installation location corresponds to data that indirectly indicates the position coordinates of the inspection point.

[0133] The flowchart shown in Figure 30 includes steps S41 to S48. Step S41 is a step in which it is determined whether the current position of the vehicle coincides with the inspection point. The agreement here is not limited to an exact match; an error of a predetermined amount, such as 0.5m or 1.0m, may be permitted. In other words, an approximate match may be permitted. Steps S42 to S48 are the same as steps S31 to S33 and S35 to S38 described above.

[0134] By the way, in the above modifications (6) and (7), a configuration was described in which the camera ECU1 performs image diagnostic processing, but the image diagnostic processing may also be performed by an external server. The camera ECU1 may periodically send concatenated image data, which is obtained by concatenating partial images acquired when an object to be inspected is detected or when an inspection point is reached, to the external server at a predetermined timing, and the server may perform image diagnostic processing based on the concatenated image data.

[0135] Furthermore, while the above examples illustrate cases where a computer such as the camera ECU1 or server automatically performs image diagnosis, image diagnosis may also be performed visually by a worker. For example, a worker may determine whether there is an abnormality in the object being inspected by comparing past linked images taken when driving the same road section with the current linked images. The inspection processing unit F47 may be configured to display the latest image and past images taken at the same location side by side on the display device 6. The functions of the inspection processing unit F47 are optional and may be omitted.

[0136] Furthermore, it is preferable that past images used for comparison with the latest image not only have similar location information but also closely match in terms of direction of travel (vehicle posture). It is also preferable to compare images where not only the position and posture of the vehicle are identical, but also the weather and time of day are consistent. By using past images with as high a degree of similarity in shooting conditions as possible as comparison points for the latest image, the inspection processing unit F47 can reduce the risk of misjudging the condition of the object being inspected.

[0137] <Additional Note> The devices, systems, and methods described in this disclosure may be implemented by a dedicated computer comprising a processor programmed to execute one or more functions embodied by a computer program. Alternatively, the devices and methods described in this disclosure may be implemented using dedicated hardware logic circuits. Furthermore, the devices and methods described in this disclosure may be implemented by one or more dedicated computers comprising a processor executing a computer program and one or more hardware logic circuits. For example, some or all of the functions of processor 11 may be implemented as hardware. Embodiments of hardware implementation include those using one or more ICs. The processor (processing core) can be a CPU, MPU, GPU, DFP (Data Flow Processor), etc. Also, some or all of the functions of processor 11 may be implemented by combining multiple types of processing units. Some or all of the functions of processor 11 may be implemented using a system-on-chip (SoC), FPGA, ASIC, etc. FPGA stands for Field-Programmable Gate Array. ASIC stands for Application Specific Integrated Circuit.

[0138] Furthermore, the computer program may be stored on a computer-readable non-transitory tangible storage medium as instructions executed by the computer. Suitable storage media for the program include HDDs (Hard-disk drives), SSDs (Solid State Drives), flash memory, etc. This disclosure also includes a computer program for operating the computer as the vehicle recording device described above, and a storage medium on which the computer program is stored. [Explanation of Symbols]

[0139] 1 Camera ECU (Vehicle Recording Device), 2 Cameras, 2A Front Camera, 2B Rear Camera, 2C Left Camera, 2D Right Camera, 3 Input Device, 4 On-board Sensor, 5 On-board ECU, 11 Processor, 12 RAM, 15 Storage Memory (Storage Medium), F1 Video Acquisition Unit, F2 Operation Reception Unit, F3 Vehicle Status Acquisition Unit, F4 Data Processing Unit, F41 Base Image Acquisition Unit (Image Acquisition Unit), F42 Region of Interest Extraction Unit (Extraction Unit), F43 Linking Processing Unit, F44 Storage Processing Unit, F45 Recognition Processing Unit, F46 Image Processing Unit, F47 Inspection Processing Unit, CR Extraction Region, PI Partial Image, CI Linked Image, CPI Camera Integrated Partial Image, RP Vehicle Horizontal Plane, Mk Marker

Claims

1. A video acquisition unit (F1) that acquires video signals from a camera installed in the vehicle, An image acquisition unit (F41) sequentially acquires image frames generated based on the video signal acquired by the video acquisition unit at a predetermined sampling interval, From the image frame acquired by the image acquisition unit, an extraction unit (F42) extracts a row range or column range set as an extraction region as a partial image (PI), A concatenation processing unit (F43) connects a plurality of partial images extracted by the extraction unit in chronological order in a direction perpendicular to the extraction direction, which is the direction in which the partial image is extracted from the image frame. A vehicle recording device comprising a storage processing unit (F44) that stores a concatenated image (CI), which is an image formed by a series of the aforementioned partial images generated by the concatenation processing unit, in a predetermined storage medium (15), The cropping portion crops the image frame vertically so that the partial image includes a predetermined column range within the image frame. The aforementioned concatenation processing unit is configured to concatenate the partial images in the horizontal direction in a vehicle recording device.

2. A vehicle recording device according to claim 1, A vehicle recording device in which the horizontal length of the aforementioned partial image is set to a constant value.

3. A video acquisition unit (F1) that acquires a video signal from a camera installed in the vehicle, An image acquisition unit (F41) sequentially acquires image frames generated based on the video signal acquired by the video acquisition unit at a predetermined sampling interval, From the image frame acquired by the image acquisition unit, an extraction unit (F42) extracts a row range or column range set as an extraction region as a partial image (PI), A concatenation processing unit (F43) connects a plurality of partial images extracted by the extraction unit in chronological order in a direction perpendicular to the extraction direction, which is the direction in which the partial image is extracted from the image frame. A vehicle recording device comprising a storage processing unit (F44) that stores a concatenated image (CI), which is an image formed by a series of the aforementioned partial images generated by the concatenation processing unit, in a predetermined storage medium (15), A recognition processing unit (F45) recognizes an object from the image formed by the aforementioned video signal, The system includes an image processing unit (F46) that generates an image by superimposing a marker (Mk), which is an image indicating the recognition state of the object, onto the image formed by the aforementioned video signal. The image acquisition unit acquires the image on which the markers are superimposed as the image frame, The cropping unit is configured to crop a portion of the area including the marker as a partial image when the marker is located on the cropping area of ​​the image frame acquired by the image acquisition unit.

4. A video acquisition unit (F1) that acquires a video signal from a camera installed in the vehicle, An image acquisition unit (F41) sequentially acquires image frames generated based on the video signal acquired by the video acquisition unit at a predetermined sampling interval, From the image frame acquired by the image acquisition unit, an extraction unit (F42) extracts a row range or column range set as an extraction region as a partial image (PI), A concatenation processing unit (F43) connects a plurality of partial images extracted by the extraction unit in chronological order in a direction perpendicular to the extraction direction, which is the direction in which the partial image is extracted from the image frame. A vehicle recording device comprising a storage processing unit (F44) that stores a concatenated image (CI), which is an image formed by a series of the aforementioned partial images generated by the concatenation processing unit, in a predetermined storage medium (15), The system includes an image processing unit (F46) that generates a processed image, which is an image obtained by applying a predetermined processing method to the image shown by the video signal. The image acquisition unit acquires the processed image as an image frame at the sampling interval, The cutting section is configured to cut out a predetermined range of the processed image as the partial image in a vehicle recording device.

5. A vehicle recording device according to claim 4, The video acquisition unit acquires the video signal from each of the multiple cameras installed to capture images of the outside of the vehicle. The image processing unit generates a composite image by combining images from multiple cameras. The image acquisition unit acquires the composite image as an image frame at the sampling interval, The cutting section cuts out a predetermined range of the composite image as the image frame as the partial image, The concatenation processing unit is configured to generate a concatenated image by arranging multiple partial images in chronological order, based on multiple composite images with different generation times.

6. A video acquisition unit (F1) that acquires a video signal from a camera installed in the vehicle, An image acquisition unit (F41) sequentially acquires image frames generated based on the video signal acquired by the video acquisition unit at a predetermined sampling interval, From the image frame acquired by the image acquisition unit, an extraction unit (F42) extracts a row range or column range set as an extraction region as a partial image (PI), A concatenation processing unit (F43) connects a plurality of partial images extracted by the extraction unit in chronological order in a direction perpendicular to the extraction direction, which is the direction in which the partial image is extracted from the image frame. A vehicle recording device comprising a storage processing unit (F44) that stores a concatenated image (CI), which is an image formed by a series of the aforementioned partial images generated by the concatenation processing unit, in a predetermined storage medium (15), The aforementioned video acquisition unit acquires the video signal from each of the multiple cameras installed in the vehicle. The image acquisition unit acquires the image frame for each camera at the sampling interval, The cutting unit generates the partial image for each camera, The aforementioned coupling processing unit is To generate a camera integrated partial image (CPI) by combining the partial images from each camera, which were taken at the same time, in a predetermined order in the cropping direction, A vehicle recording device configured to generate a concatenated image by combining the aforementioned integrated camera partial images in a time-series order in a direction perpendicular to the cutting direction.

7. A video acquisition unit (F1) that acquires a video signal from a camera installed in the vehicle, An image acquisition unit (F41) sequentially acquires image frames generated based on the video signal acquired by the video acquisition unit at a predetermined sampling interval, From the image frame acquired by the image acquisition unit, an extraction unit (F42) extracts a row range or column range set as an extraction region as a partial image (PI), A concatenation processing unit (F43) connects a plurality of partial images extracted by the extraction unit in chronological order in a direction perpendicular to the extraction direction, which is the direction in which the partial image is extracted from the image frame. A vehicle recording device comprising a storage processing unit (F44) that stores a concatenated image (CI), which is an image formed by a series of the aforementioned partial images generated by the concatenation processing unit, in a predetermined storage medium (15), The camera is mounted such that its imaging range includes a direction that is 10° or more upward from the vehicle horizontal plane (RP), which is a plane perpendicular to the vehicle's height direction. The aforementioned cropping region is set to be an area in the image frame in which the sky can be captured in a vehicle recording device.

8. A vehicle recording device according to any one of claims 1 to 7, The aforementioned sampling interval is dynamically changed according to the driving scene or speed of movement in the vehicle recording device.

9. A vehicle recording device according to any one of claims 1 to 7, The cutting unit is configured to cut out a predetermined area from the image frame in which a predetermined object to be inspected is expected to be captured, as a partial image. The aforementioned storage processing unit is To acquire data indicating the shooting conditions based on input from on-board sensors, A vehicle recording device that saves data indicating the shooting conditions in association with the partial image.

10. A vehicle recording device according to claim 9, The system includes an inspection processing unit (F47) that detects abnormalities in a predetermined object to be inspected based on the aforementioned partial image, The inspection processing unit detects an abnormality in the object to be inspected by comparing the latest image, which is the newer of the two partial images whose shooting conditions match, with the past image, which is the older of the two partial images. (Vehicle recording device)

11. A vehicle recording device according to claim 10, The inspection processing unit is, The system uses image recognition to determine whether or not the object to be inspected is visible in the partial image generated by the cutting unit. A vehicle recording device configured to attempt a comparison with past images, using the partial image as the latest image, when the partial image in which the object to be inspected is captured is generated at the cutting section.

12. A vehicle recording device according to claim 10, The system includes an inspection point storage unit, which is a storage medium on which data is registered indicating an inspection point that directly or indirectly indicates the point where the object to be inspected is reflected in the cut-out area. The inspection processing unit is, Based on the data registered in the inspection point storage unit, it is determined whether the current location is an inspection point. A vehicle recording device configured to attempt a comparison between a past image and a partial image generated at the current location, when the current location is the aforementioned inspection point, using the current location as the latest image.

13. A vehicle recording device according to claim 9, The aforementioned recording device for vehicles includes, in addition to the positional coordinates of the shooting location, at least one of the following: the presence or absence of a preceding vehicle, the size of the preceding vehicle, the type of preceding vehicle, and the distance between vehicles.

Images