Image information acquisition device, image information acquisition method, and image information acquisition program

Abstract

To compensate for image information corresponding to an abnormal camera that is in an abnormal condition with image information acquired using a normal camera that is in a normal condition.SOLUTION: An image information generating part (304) generates compensation destination image information being image information corresponding to an abnormal camera that is in an abnormal condition among a plurality of cameras, on the basis of compensation source image information being image information based on a photographic image acquired with a normal camera that is in a normal condition among the plurality of cameras. Concretely, the image information generating part generates compensation destination image information on the basis of the compensation source image information and a time difference between timing when an imaging object target has to be imaged by the abnormal camera and timing when the identical imaging object target is imaged by the normal camera.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to an image information acquisition device, an image information acquisition method, and an image information acquisition program that acquire image information based on captured images of the surroundings of a vehicle. [Background technology]

[0002] Various technologies have been proposed in the past for supplementing an image captured by one of multiple cameras whose imaging areas overlap with each other by using a normal image when the other camera is unable to capture an image due to an abnormality or malfunction. Specifically, for example, a periphery monitoring system described in Patent Document 1 includes first to fourth cameras whose imaging areas overlap with each other, and controls autonomous driving of a vehicle using the captured images. This periphery monitoring system determines whether the first to fourth cameras are normal, and if it is unable to capture an image corresponding to the imaging area of ​​any of the cameras, continues to control autonomous driving using an image captured by another camera whose imaging area overlaps. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2023-16130 Summary of the Invention [Problem to be solved by the invention]

[0004] There is still room for improvement in conventional technology (see, for example, Patent Document 1) that complements image information that would have been acquired by a malfunctioning camera with image information from another malfunctioning camera. Specifically, for example, such conventional technology is limited to complementing images captured by cameras whose imaging areas overlap. Furthermore, even if image information is complemented, applications that use such image information (e.g., monitors, vehicle control, etc.) may experience visual imbalance in the displayed image or recognition errors in image recognition. The present disclosure has been made in consideration of the circumstances exemplified above. [Means for solving the problem]

[0005] In one aspect of the present disclosure, an image information acquisition device (3) that acquires image information based on a captured image of the surroundings of a vehicle (V) includes: a camera status acquisition unit (302) that acquires whether the status of each of the plurality of cameras (2) mounted on the vehicle is a normal status in which the captured image can be acquired or an abnormal status in which the captured image cannot be acquired; an image information generating unit (304) that generates complemented image information, which is the image information corresponding to an abnormal camera among the plurality of cameras, the state of which is the abnormal state, based on complemented image information, which is the image information based on the captured image acquired by a normal camera among the plurality of cameras, the state of which is the normal state; Equipped with The image information generation unit generates the complemented image information based on the source image information and the time difference between the timing at which the same object to be photographed should be photographed by the abnormal camera and the timing at which it is photographed by the normal camera.

[0006] In another aspect of the present disclosure, an image information acquisition method executed by an image information acquisition device (3) that acquires image information based on a captured image of an area around a vehicle (V) includes: The state of each of the plurality of cameras (2) mounted on the vehicle is determined to be a normal state in which the captured image can be acquired or an abnormal state in which the captured image cannot be acquired; generating, based on the complement source image information, the image information based on the captured image acquired by a normal camera among the plurality of cameras, the state of which is the normal state, complement destination image information, the image information corresponding to the abnormal camera among the plurality of cameras, the state of which is the abnormal state; The complemented image information is generated based on the complement source image information and the time difference between the timing at which the same object to be photographed should be photographed by the abnormal camera and the timing at which it is photographed by the normal camera. In yet another aspect of the present disclosure, an image information acquisition program executed by an image information acquisition device (3) that acquires image information based on a captured image of a surrounding area of ​​a vehicle (V) includes: The process executed by the image information acquisition device includes: A process of acquiring whether the state of each of the plurality of cameras (2) mounted on the vehicle is a normal state in which the captured image can be acquired or an abnormal state in which the captured image cannot be acquired; generating, based on complement source image information, the image information based on the captured image acquired by a normal camera of the plurality of cameras whose state is the normal state, complement destination image information, the image information corresponding to the abnormal camera of the plurality of cameras whose state is the abnormal state; Including, The process of generating the complemented image information is based on the source image information and the time difference between the timing at which the same object to be photographed should be photographed by the abnormal camera and the timing at which it is photographed by the normal camera.

[0007] In addition, in each section of the application documents, each element may be assigned a reference symbol in parentheses. However, such reference symbols merely indicate an example of the correspondence between the element and the specific means described in the embodiments below. Therefore, the present disclosure is not limited in any way by the above-mentioned reference symbols. [Brief explanation of the drawings]

[0008] [Figure 1]1 is a block diagram illustrating a schematic configuration of an in-vehicle system including an image information acquisition device according to an embodiment of the present disclosure. [Figure 2] 2 is a block diagram showing a schematic functional configuration realized in the image information acquisition device shown in FIG. 1. FIG. [Figure 3] 2 is a schematic diagram showing a state in which a vehicle equipped with the in-vehicle system shown in FIG. 1 is traveling. [Figure 4] 3 is a flowchart showing an outline of an example of the operation of the image information acquisition device shown in FIG. 2. [Figure 5] 2 is a schematic diagram showing a state in which a vehicle equipped with a modified example of the in-vehicle system shown in FIG. 1 is traveling. DETAILED DESCRIPTION OF THE INVENTION

[0009] (Embodiment) Hereinafter, exemplary embodiments and specific examples of the present disclosure will be described with reference to the drawings as appropriate. First, referring to FIG. 1, an in-vehicle system 1 is configured to be mounted on a vehicle and to perform various operations in the vehicle. Hereinafter, the vehicle equipped with the in-vehicle system 1 will be referred to as "host vehicle V" (see FIG. 3). In this embodiment, the in-vehicle system 1 is equipped with multiple cameras 2 that capture images of the surroundings of the host vehicle V, and is configured to use images captured by these cameras 2 to display information to occupants of the host vehicle V, record the information, perform ADAS operations, and perform various other operations. ADAS is an abbreviation for Advanced Driver-Assistance Systems.

[0010] The in-vehicle system 1 includes a camera 2, an image information acquisition device 3, an in-vehicle sensor 4, a monitor 5, and a vehicle control ECU 6. ECU stands for Electronic Control Unit. The camera 2 and the image information acquisition device 3 are connected to each other so that they can exchange information or signals. The image information acquisition device 3 is connected to the in-vehicle sensor 4, the monitor 5, and the vehicle control ECU 6 via an in-vehicle network so that they can exchange information or signals with each other. The in-vehicle network is configured to comply with a predetermined communication standard such as CAN (international registered trademark: international registration number 1048262A). CAN (international registered trademark) is an abbreviation for Controller Area Network. Note that the in-vehicle network may have a sub-network that complies with LIN, FlexRay, or the like, in addition to a main network that complies with CAN (international registered trademark). LIN is an abbreviation for Local Interconnect Network. The following describes the schematic configuration and function of each part of the in-vehicle system 1.

[0011] The camera 2 is equipped with an image sensor such as a CCD or CMOS, and is mounted at a predetermined position on the host vehicle V to capture images of targets around the host vehicle V. CCD stands for Charge Coupled Device. CMOS stands for Complementary Metal Oxide Semiconductor. "Targets" include three-dimensional objects that may be obstacles to travel, such as pedestrians and other vehicles, as well as traffic lights, road signs, and road markings. In this embodiment, the in-vehicle system 1 is equipped with at least a front camera 21, a right front side camera 22, and a left front side camera 23 as the multiple cameras 2. The front camera 21 is provided so as to capture an image of the area in front of the host vehicle V. The right front side camera 22 is provided so as to capture an image of the area diagonally forward to the right and to the right of the host vehicle V. The left front side camera 23 is provided so as to capture an image of the area diagonally forward to the left and to the left of the host vehicle V.

[0012] The image information acquisition device 3 is configured to acquire image information for displaying on the monitor 5 and for performing image recognition for driving control of the host vehicle V in the vehicle control ECU 6, based on an image captured by the camera 2 of the surroundings of the host vehicle V. In this embodiment, the image information acquisition device 3 is configured as an image processing ECU or an image recognition ECU, which is an on-board computer. That is, the image information acquisition device 3 includes a processor 31 and a storage medium 32 connected to the processor 31 so as to be able to communicate information. The image information acquisition device 3 is configured so that the processor 31 reads and executes a computer program from the storage medium 32, thereby realizing a predetermined function for displaying images or recognizing objects around the host vehicle V.

[0013] The processor 31 includes at least one arithmetic unit configured as a CPU or MPU and its peripheral circuits (e.g., a timer circuit, etc.). The storage medium 32 includes at least a ROM or a nonvolatile rewritable memory among various non-transient physical storage media such as a ROM or a nonvolatile rewritable memory. A nonvolatile rewritable memory is a storage device that allows information to be rewritten while the power is on but retains information in an unrewritable manner while the power is off, such as a flash memory. The storage medium 32 stores the above-mentioned computer program as well as various data such as initial values, maps, and look-up tables required to execute the program. Further details of the functions of the image information acquisition device 3 will be described later.

[0014] The on-board sensor 4 is configured to detect various quantities related to the driving state of the host vehicle V. The "driving state" includes the driving operation state, driving behavior state, and driving environment state of the host vehicle V. The "driving operation state" refers to a state related to the driving operation input of the host vehicle V by the driver of the host vehicle V or the vehicle control ECU 6, and includes, for example, the steering amount, throttle opening, brake operation amount, shift range, etc. The "driving behavior state" refers to a state related to the movement or behavior of the host vehicle V, and includes, for example, the vehicle speed, acceleration, yaw rate, etc. The "driving environment state" refers to the environment around the host vehicle V other than the state of the target to be photographed by the camera 2, and includes, for example, the illuminance, weather, outside temperature, road surface condition, etc. around the host vehicle V.

[0015] The monitor 5 has at least one display device, such as an organic EL panel or a liquid crystal panel, for displaying image information based on images captured by the camera 2. EL stands for Electro Luminescence. An organic EL panel may also be called OLED. OLED stands for Organic Light Emitting Diode. The monitor 5 is provided in the cabin of the vehicle V so that it can be easily viewed by a passenger (e.g., the driver).

[0016] The vehicle control ECU 6 is configured as a so-called driving ECU, which is an on-board computer that controls the driving force generation mechanism, driving force transmission mechanism, braking mechanism, steering mechanism, etc. of the host vehicle V. Specifically, in this embodiment, the vehicle control ECU 6 is a so-called driving automation ECU that is configured to realize a driving automation function in the host vehicle V. "Driving automation" refers to realizing any of driving automation levels 1 to 5 specified in the standard "SAE J3016" published by SAE International. In other words, "driving automation" includes "driving assistance" and "automated driving." "Driving assistance" includes "driving assistance" in the narrow sense, in which both longitudinal vehicle motion control subtasks and lateral vehicle motion control subtasks are not executed simultaneously, and "advanced driving assistance," in which both are executed simultaneously. The longitudinal vehicle motion control subtasks are starting, accelerating / decelerating, and stopping. The lateral vehicle motion control subtask is steering.

[0017] (Image information acquisition device) 2, the image information acquisition device 3 includes, as functional components realized on an on-board microcomputer by execution of a computer program, a captured image acquisition unit 301, a camera state acquisition unit 302, an image processing unit 303, an image information generation unit 304, and a recognition processing unit 305. FIG. 3 shows the state of the host vehicle V while it is traveling. Hereinafter, the details of each functional component of the image information acquisition device 3 will be described with reference to FIG. 2 and FIG. 3 in addition to FIG. 1.

[0018] The captured image acquisition unit 301 is configured to acquire captured image information from each of the multiple cameras 2. The camera state acquisition unit 302 is configured to acquire the camera state, i.e., whether the state of each of the multiple cameras 2 mounted on the vehicle V is a normal state in which captured images can be acquired, or an abnormal state in which captured images cannot be acquired. Specifically, in this embodiment, the camera state acquisition unit 302 is configured to determine the camera state of the camera 2 based on input information including captured image information from the camera 2. The image processing unit 303 is configured to perform various well-known image processing such as brightness correction on the acquired captured image information.

[0019] The image information generation unit 304 generates image information to be displayed on the monitor 5 and used for image recognition by the recognition processing unit 305, based on the captured image information after image processing. The image information generation unit 304 also complements image information corresponding to an abnormal camera, whose camera state is abnormal, with image information acquired using a normal camera, whose camera state is normal. Image information corresponding to an abnormal camera, i.e., image information that is estimated to have been acquired if the abnormal camera had been in a normal state, is referred to as "complemented image information." In contrast, image information based on an image captured by a normal camera and used as the basis for generating the complemented image information is referred to as "source image information." The operation of generating the complemented image information based on the source image information is referred to as an image complement operation.

[0020] An overview of the image complementing operation by the image information generating unit 304 in this embodiment will be described below with reference to FIG. 3 in addition to FIG. 1 and FIG. 2. In FIG. 3, the center line L is an extension of the vehicle center line of the host vehicle V in the traveling direction. The forward imaging area R1 indicates the imaging area captured by the forward camera 21. The right front side imaging area R2 indicates the imaging area captured by the right front side camera 22. The left front side imaging area R3 indicates the imaging area captured by the left front side camera 23. Note that in FIG. 3, the forward imaging area R1, the right front side imaging area R2, and the left front side imaging area R3 are illustrated as extending from a position on the center line L in front of the host vehicle V, but this is merely for the sake of simplification of the illustration, i.e., for ease of viewing. Therefore, in reality, the forward camera 21, the right front side camera 22, and the left front side camera 23 may be mounted at different positions in the vehicle width direction and the vehicle length direction. The same applies to FIG. 5, which corresponds to a modified example described later.

[0021] In this embodiment, as shown in Fig. 3, the front imaging region R1 does not include an overlapping region between the right front side imaging region R2 and the left front side imaging region R3. That is, the imaging region of the front camera 21 does not overlap with the imaging regions of the right front side camera 22 and the left front side camera 23. Similarly, the right front side imaging region R2 and the left front side imaging region R3 do not include an overlapping region with each other. In such a camera arrangement in which the imaging regions do not overlap with each other, image information complementation techniques using overlapping imaging regions, as in the prior art, cannot perform a satisfactory image complementation operation.

[0022] Therefore, the image information generation unit 304 calculates the time difference between the timing when the same object to be photographed should be photographed by the abnormal camera and the timing when the object is photographed by the normal camera. This time difference can be calculated based on the camera arrangement and the driving state of the host vehicle V, including the traveling speed. Then, the image information generation unit 304 generates the complemented image information based on the calculated time difference and the source image information. That is, the image information generation unit 304 acquires the complemented image information by adding a time difference to the source image information.

[0023] Specifically, for example, referring to FIG. 3, assume that the right front-side camera 22 is an abnormal camera and the front camera 21 is a normal camera. In this case, the time difference is the difference in the timing of capturing images. More specifically, a single line-shaped paint mark on a broken road dividing line W in an area to the right of the center line L in the forward imaging area R1 is defined as the source image M1. A time difference occurs between the time when the source image M1 moves to the position of the destination image M3 in the right front-side imaging area R2, depending on the camera arrangement and the driving conditions of the vehicle V, such as the vehicle speed and yaw rate. For this reason, the image information generating unit 304 does not immediately generate image information including the source image M1 as the destination image information, but rather delays the time difference to generate the destination image information. This allows the monitor 5 to seamlessly display image information based on the image captured by the left front-side camera 23 (in a normal state) and image information based on the destination image information. Furthermore, a decrease in the recognition accuracy of the road dividing line W on the right side of the vehicle V can be effectively suppressed.

[0024] (Example of operation) Below, with reference to the flowchart shown in Fig. 4 in addition to Figs. 1 to 3, an overview of the image complementing operation by the image information acquisition device 3 according to this embodiment will be described, along with the effects achieved by the image information acquisition method and image information acquisition program executed by this device. In the flowchart shown in Fig. 4, "S" is an abbreviation for "step." Furthermore, the image information acquisition device 3 according to this embodiment and the image information acquisition method and image information acquisition program executed thereby may hereinafter be collectively referred to as "this embodiment."

[0025] The flowchart shown in Fig. 4 shows a process for generating, i.e., acquiring, right front side image information, which is image information corresponding to the right front side imaging area R2, i.e., the right front side camera 22. The processor 31 in the image information acquisition device 3 executes the right front side image generation routine shown in Fig. 4 at predetermined time intervals (e.g., 10 msec) while a predetermined execution condition (e.g., shift range, etc.) is met.

[0026] When this routine is executed, first, in step 101, the processor 31 acquires captured image information from each of the multiple cameras 2. Next, in step 102, the processor 31 performs predetermined image processing on the acquired captured image information. Subsequently, in step 103, the processor 31 determines whether the right front side camera 22 is in a normal state or an abnormal state.

[0027] If the right front-side camera 22 is in a normal state (i.e., step 103=YES), the processor 31 executes the processing of step 104 and then temporarily ends this routine. In step 104, the processor 31 generates right front-side image information based on the image actually captured by the right front-side camera 22.

[0028] If the right front side camera 22 is in an abnormal state (i.e., step 103=NO), the processor 31 proceeds to step 105. In step 105, the processor 31 determines whether the front camera 21 is in a normal state.

[0029] If the front camera 21 is in a normal state (i.e., step 105=YES), the processor 31 executes the processes of steps 106 and 107, and then temporarily ends this routine. In step 106, the processor 31 generates right front side provisional image information based on the image captured by the front camera 21. In step 107, the processor 31 calculates the above-mentioned time difference and generates right front side image information based on this time difference and the right front side provisional image information. In this way, right front side image information obtained by delaying the right front side provisional image information by the above-mentioned time difference is acquired. On the other hand, if the front camera 21 is in an abnormal state (i.e., step 105=NO), the processor 31 skips the processes of steps 106 and 107 and temporarily ends this routine.

[0030] As described above, according to this embodiment, as long as the object to be photographed is the same, image complementation is possible regardless of whether the imaging ranges overlap, and without causing problems such as visual imbalance in the displayed image or recognition errors in image recognition. Therefore, according to this embodiment, it is possible to satisfactorily complement image information corresponding to an abnormal camera whose camera condition is abnormal with image information acquired using a normal camera whose camera condition is normal.

[0031] (Variation) The present disclosure is not limited to the above-described embodiments and specific examples. Therefore, the above-described embodiments and the like can be modified as appropriate. Representative modifications will be described below. In the following description of the modifications, differences from the above-described embodiments and the like will be mainly described. Furthermore, the same reference numerals are used for parts that are identical or equivalent to each other in the above-described embodiments and the following modifications. Therefore, in the following description of the modifications, the explanations in the above-described embodiments and the like can be used as appropriate for components that have the same reference numerals as the above-described embodiments and the like, unless there is a technical contradiction or special additional explanation.

[0032] The present disclosure is not limited to the specific applications and device configurations shown in the above embodiments. That is, for example, the host vehicle V may be a so-called automobile or a motorcycle. There are no particular limitations on the type of automobile or motorcycle. There are also no particular limitations on the application of the acquired image information, and it can be used for, for example, an around-view monitor, a blind-spot monitor, an electronic side mirror, white line recognition, road sign recognition, pedestrian recognition, etc.

[0033] The present disclosure may also be suitably applied to an embodiment in which the forward imaging region R1 includes an overlapping region between the right front-side imaging region R2 and the left front-side imaging region R3, as shown in FIG. 5 . Furthermore, in the above embodiment, for the sake of simplicity of illustration and explanation, the front camera 21, the right front-side camera 22, and the left front-side camera 23 are illustrated as the multiple cameras 2, but the present disclosure is not limited to such an embodiment. That is, for example, when acquiring image information in front of and on the sides of the host vehicle V, other cameras (e.g., a front wide-angle camera) may also be provided as the multiple cameras 2. The present disclosure may also be suitably applied to the acquisition of image information behind and on the sides of the host vehicle V using, for example, a rear camera, a right rear-side camera, and a left rear-side camera.

[0034] All or part of the image information acquisition device 3 may be configured with a digital circuit, such as an ASIC or FPGA, configured to be able to realize the above-mentioned functions or operations. ASIC stands for Application Specific Integrated Circuit. FPGA stands for Field Programmable Gate Array. In other words, the image information acquisition device 3 may have both an on-board microcomputer and a digital circuit.

[0035] A computer program according to the present disclosure that enables the execution of various operations, procedures, or processes described in the above embodiments can be downloaded or upgraded via V2X communication. V2X stands for Vehicle to X. Alternatively, such a computer program can be downloaded or upgraded via a terminal device installed in a manufacturing plant, a repair shop, a dealer, or the like of the vehicle V. Such a computer program can be stored on a memory card, an optical disk, a magnetic disk, or the like.

[0036] In this way, each of the above functional configurations and processes may be realized by a special-purpose computer provided by configuring a processor 31 and a storage medium 32 programmed to execute one or more functions embodied in a computer program. Alternatively, each of the above functional configurations and processes may be realized by a special-purpose computer provided by configuring a processor 31 with one or more dedicated hardware logic circuits. Alternatively, each of the above functional configurations and processes may be realized by one or more special-purpose computers configured by combining one or more processors 31 programmed to execute one or more functions and one or more storage media 32 with one or more other processors 31 configured with one or more hardware logic circuits. Furthermore, a computer program may be stored in a computer-readable, non-transitory storage medium as instructions to be executed by a computer. In other words, each of the above functional configurations and processes may be expressed as a computer program including procedures for implementing the same, or as a non-transitory storage medium storing the computer program.

[0037] The present disclosure is not limited to the specific functions and operational aspects described in the above embodiment. For example, the camera status acquisition unit 302 may acquire the camera status of each of the multiple cameras 2 by receiving the camera status as a self-diagnosis result from the camera ECU provided in each of the multiple cameras 2 from the camera ECU. Also, in FIG. 4 , if there is a camera 2 other than the front camera 21 that can complement the right front side image information, the process performed when the determination result in step 105 is "NO" may involve determining the possibility of complementation by the camera 2 and executing image complementation processing if possible. Furthermore, for the above embodiment, a specific example of generating right front side image information is described for the sake of simplicity of illustration and explanation. However, the present disclosure is not limited to such an aspect. That is, the same process may be used to generate left front side image information corresponding to the case where the left front side camera 23 is an abnormal camera and the front camera 21 is a normal camera.

[0038] Similar expressions such as "acquire," "calculate," "estimate," "detect," and "sensing" may be substituted for each other as appropriate within the scope of technical inconsistency. Furthermore, "exceeding the threshold" and "above the threshold" may be substituted for each other as appropriate within the scope of technical inconsistency. The same applies to "below the threshold" and "below the threshold."

[0039] It goes without saying that the elements constituting the above-described embodiments are not necessarily essential unless expressly stated as essential or clearly considered essential in principle. Furthermore, when numerical values ​​such as the number, value, amount, and range of components are mentioned, the present disclosure is not limited to those specific numbers unless expressly stated as essential or clearly limited to a specific number in principle. Similarly, when the shape, direction, positional relationship, etc. of components are mentioned, the present disclosure is not limited to those shapes, directions, positional relationships, etc. unless expressly stated as essential or clearly limited to a specific shape, direction, positional relationship, etc. in principle.

[0040] The modified examples are not limited to the above examples. For example, all or part of one of the multiple specific examples may be combined with all or part of another of the multiple specific examples, provided that there is no technical inconsistency. There is no particular limit to the number of combinations. Similarly, all or part of one of the multiple modified examples may be combined with all or part of another of the multiple modified examples, provided that there is no technical inconsistency. Furthermore, all or part of the above specific example and all or part of the above modified examples may be combined with each other, provided that there is no technical inconsistency.

[0041] (Disclosure perspective) As is clear from the above description of the embodiments and modifications, this specification discloses at least the following matters.

[0042] [Point 1-1] An image information acquisition device (3) that acquires image information based on a captured image of the surroundings of a vehicle (V), a camera status acquisition unit (302) that acquires whether the status of each of the plurality of cameras (2) mounted on the vehicle is a normal status in which the captured image can be acquired or an abnormal status in which the captured image cannot be acquired; an image information generating unit (304) that generates complemented image information, which is the image information corresponding to an abnormal camera among the plurality of cameras, the state of which is the abnormal state, based on complemented image information, which is the image information based on the captured image acquired by a normal camera among the plurality of cameras, the state of which is the normal state; Equipped with the image information generating unit generates the complemented image information based on the complement source image information and a time difference between a timing at which the same photographing target object should be photographed by the abnormal camera and a timing at which the same photographing target object is photographed by the normal camera; Image information acquisition device. [Point 1-2] the image information generation unit calculates the time difference based on a driving state including a traveling speed of the vehicle. The image information acquisition device according to aspect 1-1. [Points 1-3] The abnormal camera and the normal camera are provided so that their imaging areas do not overlap. The image information acquisition device according to Aspect 1-1 or Aspect 1-2. [Points 1-4] The object to be photographed is a road dividing line (W), The normal camera is a front camera (21), The abnormal camera is a right front side camera (22) or a left front side camera (23). The image information acquisition device according to any one of Aspects 1-1 to 1-3.

[0043] [Point 2-1] An image information acquisition method executed by an image information acquisition device (3) that acquires image information based on captured images of the surroundings of a vehicle (V), The state of each of the plurality of cameras (2) mounted on the vehicle is determined to be a normal state in which the captured image can be acquired or an abnormal state in which the captured image cannot be acquired; generating, based on the complement source image information, the image information based on the captured image acquired by a normal camera among the plurality of cameras, the state of which is the normal state, complement destination image information, the image information corresponding to the abnormal camera among the plurality of cameras, the state of which is the abnormal state; The interpolated image information is generated based on the interpolated image information and a time difference between the timing at which the same object to be photographed should be photographed by the abnormal camera and the timing at which the object is photographed by the normal camera. Image information acquisition method. [Point 2-2] calculating the time difference based on a driving state including a traveling speed of the vehicle; The image information acquisition method according to aspect 2-1. [Point 2-3] The abnormal camera and the normal camera are provided so that their imaging areas do not overlap. The image information acquisition method according to aspect 2-1 or aspect 2-2. [Point 2-4] The object to be photographed is a road dividing line (W), The normal camera is a front camera (21), The abnormal camera is a right front side camera (22) or a left front side camera (23). The image information acquisition method according to any one of Aspects 2-1 to 2-3.

[0044] [Point 3-1] An image information acquisition program executed by an image information acquisition device (3) that acquires image information based on a captured image of the surroundings of a vehicle (V), The process executed by the image information acquisition device is A process of acquiring whether the state of each of the plurality of cameras (2) mounted on the vehicle is a normal state in which the captured image can be acquired or an abnormal state in which the captured image cannot be acquired; generating, based on complement source image information, the image information based on the captured image acquired by a normal camera of the plurality of cameras whose state is the normal state, complement destination image information, the image information corresponding to the abnormal camera of the plurality of cameras whose state is the abnormal state; Including, The process of generating the complemented image information is based on the complemented image information and a time difference between the timing at which the same object to be photographed should be photographed by the abnormal camera and the timing at which the object is photographed by the normal camera. Image information acquisition program. [Point 3-2] calculating the time difference based on a driving state including a traveling speed of the vehicle; An image information acquisition program according to aspect 3-1. [Point 3-3] The abnormal camera and the normal camera are provided so that their imaging areas do not overlap. An image information acquisition program according to aspect 3-1 or aspect 3-2. [Point 3-4] The object to be photographed is a road dividing line (W), The normal camera is a front camera (21), The abnormal camera is a right front side camera (22) or a left front side camera (23). The image information acquisition program according to any one of Aspects 3-1 to 3-3. [Explanation of symbols]

[0045] 2 Cameras 21 Front camera 22 Right front side camera 23 Left front side camera 3. Image information acquisition device 302 Camera status acquisition unit 304 Image information generation unit R1 forward imaging area R2 Right anterior lateral imaging area V Vehicle

Claims

1. An image information acquisition device (3) that acquires image information based on a captured image of the surroundings of a vehicle (V), a camera status acquisition unit (302) that acquires whether the status of each of the plurality of cameras (2) mounted on the vehicle is a normal status in which the captured image can be acquired or an abnormal status in which the captured image cannot be acquired; an image information generating unit (304) that generates complemented image information, which is the image information corresponding to an abnormal camera among the plurality of cameras, the state of which is the abnormal state, based on complemented image information, which is the image information based on the captured image acquired by a normal camera among the plurality of cameras, the state of which is the normal state; Equipped with the image information generating unit generates the complemented image information based on the complement source image information and a time difference between a timing at which the same photographing target object should be photographed by the abnormal camera and a timing at which the same photographing target object is photographed by the normal camera; Image information acquisition device.

2. the image information generation unit calculates the time difference based on a driving state including a traveling speed of the vehicle. The image information acquisition device according to claim 1 .

3. The abnormal camera and the normal camera are provided so that their imaging areas do not overlap.

3. The image information acquisition device according to claim 1.

4. The object to be photographed is a road dividing line (W), The normal camera is a front camera (21), The abnormal camera is a right front side camera (22) or a left front side camera (23).

3. The image information acquisition device according to claim 1.

5. An image information acquisition method executed by an image information acquisition device (3) that acquires image information based on captured images of the surroundings of a vehicle (V), comprising: The state of each of the plurality of cameras (2) mounted on the vehicle is determined to be a normal state in which the captured image can be acquired or an abnormal state in which the captured image cannot be acquired; generating, based on the complement source image information, the image information based on the captured image acquired by a normal camera among the plurality of cameras, the state of which is the normal state, complement destination image information, the image information corresponding to the abnormal camera among the plurality of cameras, the state of which is the abnormal state; The interpolated image information is generated based on the interpolated image information and a time difference between the timing at which the same object to be photographed should be photographed by the abnormal camera and the timing at which the object is photographed by the normal camera. Image information acquisition method.

6. An image information acquisition program executed by an image information acquisition device (3) that acquires image information based on a captured image of the surroundings of a vehicle (V), The process executed by the image information acquisition device is A process of acquiring whether the state of each of the plurality of cameras (2) mounted on the vehicle is a normal state in which the captured image can be acquired or an abnormal state in which the captured image cannot be acquired; generating, based on complement source image information, the image information based on the captured image acquired by a normal camera of the plurality of cameras whose state is the normal state, complement destination image information, the image information corresponding to the abnormal camera of the plurality of cameras whose state is the abnormal state; Including, The process of generating the complemented image information is based on the complemented image information and a time difference between the timing at which the same object to be photographed should be photographed by the abnormal camera and the timing at which the object is photographed by the normal camera. Image information acquisition program.

Images