In-vehicle camera device, in-vehicle camera system, and image storage method

Abstract

To provide an on-vehicle camera apparatus which automatically extracts images before and after occurence of erroneous recognition of an external environment from images captured under an actually traveling environment.SOLUTION: The present invention is directed to an on-vehicle camera apparatus for discriminating an object. The on-vehicle camera apparatus includes: a control feed-back unit for receiving a feedback of automatic control of own vehicle; an erroneous control determination unit for determining erroneous automatic control based on drive operation information of a driver; and an image storing unit for storing images, wherein images are stored when the erroneous automatic control is determined.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an in-vehicle camera device, an in-vehicle camera system, and an image storage method that generate learning data for machine learning from images captured while driving. [Background technology]

[0002] In recent years, vehicles equipped with autonomous driving (AD) systems and advanced driver-assistance systems (ADAS) have become increasingly common. To realize these systems, it is necessary to accurately recognize the vehicle's external environment (other vehicles, pedestrians, bicycles, stationary obstacles, and their positions, speeds, and moving directions). To this end, image recognition processing is performed on the output data of the vehicle's onboard camera to recognize the vehicle's external environment. Some of the image recognition processing uses machine learning.

[0003] Machine learning uses data from diverse situations for learning. Increasing the diversity of data used in machine learning (hereinafter referred to as "learning data") can improve the recognition accuracy of an AI network that has learned this data through machine learning. One known technology for increasing the diversity of learning data is a learning data generation device described in Patent Document 1, which acquires learning data from output data from on-board sensors based on the driver's manual driving operations.

[0004] The abstract of Patent Document 1 states that the problem is to "generate learning data for realizing highly accurate driving operations during automated driving based on driving operations during manual driving," and as a solution to this problem, it states that "the learning data generation device comprises a memory unit that stores various data, a display unit that can display images, a driving environment acquisition unit that acquires information about the vehicle's driving environment, an operation acquisition unit that acquires inputs related to vehicle operation, and a control unit that generates learning data that associates the information about the driving environment with the inputs related to the operations, and the control unit displays the information about the driving environment acquired by the driving environment acquisition unit on the display unit, and generates learning data that associates the inputs related to the operations acquired by the operation acquisition unit with the displayed information about the driving environment." [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2020-160513 Summary of the Invention [Problem to be solved by the invention]

[0006] The learning data generation device in Patent Document 1 has the ultimate goal of realizing highly accurate autonomous driving, as stated in the problem section of the abstract: "The device generates learning data to enable highly accurate driving operations during autonomous driving."

[0007] However, as stated in paragraph 0022 of Patent Document 1, "The simulator system (learning data generation device) 1 is a system in which a vehicle travels in a virtually constructed driving environment and acquires the driving operations of the driver while traveling," the learning data generation device of Patent Document 1 is merely a device that uses the manual driving operations of a driver in a given virtually constructed driving environment as learning data, and does not use acquired data from various situations that occur in a real driving environment as learning data. Therefore, it is not possible to learn situations that the simulator designer or operator cannot fully anticipate, and there is a possibility that the accuracy of autonomous driving cannot be fully guaranteed in unlearned situations.

[0008] On the other hand, another possible learning method is to increase the amount of training data by using sensor outputs generated in actual driving environments as training data, and have the AI ​​network learn various driving environments by machine learning. In this case, it is easy to generate a huge amount of training data, but this learning method is also not realistic considering the following: a huge amount of memory capacity is required to store the huge amount of training data, the annotation work to assign correct labels to the training data is also enormous, and furthermore, most of the huge amount of training data is training data for normal recognition that does not contribute to improving the AI ​​network, and most of the machine learning is invalid learning.

[0009] If it were possible to exclude learning data that does not contribute to improving the AI ​​network from the vast amount of learning data generated under real driving environments and extract only the learning data that does contribute to improvement, it would be possible to suppress ineffective learning during machine learning of the AI ​​network; however, no specific proposals have been made so far regarding how to extract such learning data.

[0010] Therefore, the present invention aims to provide an in-vehicle camera device, an in-vehicle camera system, and an image storage method that automatically extract and store images before and after an erroneous recognition of the external environment from various images captured in a real driving environment, thereby contributing to the improvement of problematic AI networks. [Means for solving the problem]

[0011] In order to solve the above problem, the vehicle-mounted camera device of the present invention is equipped with a control feedback unit that receives feedback on the automatic control of the vehicle itself, an erroneous control judgment unit that judges erroneous automatic control based on the driver's driving operation information, and an image storage unit that saves images, and saves images when erroneous automatic control is judged to be occurring. [Effects of the Invention]

[0012] The present invention makes it possible to automatically extract and save images taken before and after a misrecognition of the external environment from various images captured in a real driving environment, which contributes to improving problematic AI networks. Other problems, configurations, and effects will become clear from the following description of the preferred embodiment of the invention. [Brief explanation of the drawings]

[0013] [Figure 1] 3 is a bird's-eye view illustrating the field of view of an in-vehicle camera device mounted on the vehicle; [Figure 2] FIG. 10 is a diagram showing the correlation between each visual field area. [Figure 3] FIG. 1 is a functional block diagram of an in-vehicle camera system according to an embodiment. [Figure 4] 10 is a bird's-eye view illustrating the behavior of the host vehicle when an erroneous identification occurs. [Figure 5] 10 is a flowchart illustrating a process for saving images before and after misclassification. [Figure 6A] FIG. 10 is a bird's-eye view illustrating the behavior of a vehicle when erroneous tracking occurs. [Figure 6B] Legend for Figure 6A. [Figure 7] 10 is a flowchart illustrating a process for saving images before and after erroneous tracking. [Figure 8] 10 is a bird's-eye view illustrating the behavior of the host vehicle when an erroneous identification occurs. [Figure 9] 10 is a flowchart illustrating a process for saving images before and after misclassification. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, with reference to FIGS. 1 to 7, an embodiment of an in-vehicle camera device 100 of the present invention and an in-vehicle camera system including the device will be described.

[0015] The in-vehicle camera device 100 according to this embodiment is a device that identifies and saves images that have caused AD control or ADAS control that is contrary to the driver's intention, taking into account the details of the driver's manual driving operation, and uses the saved images to perform additional learning on an AI network for recognizing the external environment. The in-vehicle camera system according to this embodiment is a system that includes the in-vehicle camera device 100 and a vehicle control system 200 that controls the vehicle based on the output of the in-vehicle camera device 100. Each of these will be described in detail below.

[0016] FIG. 1 is a plan view illustrating the field of view angle of an in-vehicle camera device 100 mounted on a vehicle. As shown in FIG. 1, the in-vehicle camera device 100 is attached facing forward on the inside of the windshield of the vehicle, and incorporates a left image capturing unit 1L and a right image capturing unit 1R. The left image capturing unit 1L is disposed on the left side of the in-vehicle camera device 100 and captures a left image P L The imaging unit is for capturing an image of the object, and its optical axis A L is facing forward of the vehicle, and the light axis A L Right field of view V is wider on the right side R The right imaging unit 1R is disposed on the right side of the vehicle-mounted camera device 100 and captures a right image P R The imaging unit is for capturing an image of the object, and its optical axis A R is facing forward of the vehicle, and the light axis A R Left field of view V is wider on the left side L It has.

[0017] As shown in the figure, the right visual field V R and left visual field V L The overlapping visual field areas are referred to below as the "stereo visual field area R S " and the right visual field V R from the stereo field of view R S The visual field excluding the right monocular visual field R R " and the left visual field V L from the stereo field of view RS The visual field excluding the left monocular visual field R L " is called.

[0018] 2 is a diagram showing the correlation between the above-mentioned fields of view. The top row shows the left image P captured by the left imaging unit 1L. L The middle row shows the right image P captured by the right imaging unit 1R. R , the bottom row is the left image P L and the right image P R A composite image P C The composite image P C As shown in the figure, the right monocular visual field area R R and the stereo field of view R S and left monocular visual field area R L It can be divided into:

[0019] 3 is a functional block diagram of the vehicle-mounted camera system according to this embodiment. As shown in this figure, the vehicle-mounted camera system according to this embodiment is a system including a vehicle-mounted camera device 100 and a vehicle control system 200, and receives output data from each sensor described below.

[0020] In addition to the left imaging unit 1L and right imaging unit 1R described above, the vehicle-mounted camera device 100 is equipped with a stereo matching unit 2, a monocular detection unit 3, a monocular ranging unit 4, a template creation unit 5, an image memory unit 6, a similar location search unit 7, a field of view identification unit 8, a stereo detection unit 9, a speed calculation unit 10, a vehicle information input unit 11, a stereo ranging unit 12, a type identification unit 13, a driver operation input unit 14, an identification history memory unit 15, a tracking history memory unit 16, a position history memory unit 17, a speed history memory unit 18, an incorrect identification identification unit 19, an incorrect tracking identification unit 20, an incorrect control judgment unit 21, an incorrectly recognized image storage unit 22, a control feedback unit 23, and an additional learning unit 24. In addition, the vehicle-mounted camera device 100 is connected to a vehicle speed sensor 31, a vehicle steering angle sensor 32, a yaw rate sensor 33, a steering sensor 34, an accelerator pedal sensor 35, and a brake pedal sensor 36 mounted on the vehicle, and receives output data from each sensor.

[0021] The components of the vehicle-mounted camera device 100, excluding the left and right imaging units 1L and 1R, are specifically a computer equipped with hardware such as an arithmetic unit, a storage device, a communication device, etc. An arithmetic unit such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), or a CPLD (Complex Programmable Logic Device) executes a predetermined program acquired from a program recording medium or a distribution server outside the vehicle to realize each processing unit such as the stereo matching unit 2, and a storage device such as a semiconductor memory realizes each storage unit such as the identification history storage unit 15. However, in the following, such well-known technologies in the computer field will be omitted as appropriate.

[0022] The vehicle control system 200 is connected to the vehicle-mounted camera device 100 and the above-mentioned sensors, and is a system that automatically controls the vehicle's alarm, braking, and steering based on the recognition results of the vehicle-mounted camera device 100, for purposes such as collision avoidance or mitigation, following the vehicle ahead, and maintaining lane travel.

[0023] <Details of the vehicle-mounted camera device 100> Hereinafter, each part of the vehicle-mounted camera device 100 of this embodiment will be described in detail.

[0024] The left imaging unit 1L and the right imaging unit 1R are monocular cameras equipped with an imaging sensor (such as a CMOS (Complementary Metal Oxide Semiconductor)) that converts light into an electrical signal. The information converted into an electrical signal by each imaging sensor is further converted into image data representing the captured image within each imaging unit. The images captured by the left imaging unit 1L and the right imaging unit 1R are called left images P L and right image P Rand transmitted to the stereo matching unit 2, monocular detection unit 3, monocular distance measurement unit 4, template creation unit 5, image storage unit 6, similar part search unit 7, and field of view specification unit 8 at a predetermined cycle (for example, every 17 ms).

[0025] The stereo matching unit 2 receives data including image data from the left image capturing unit 1L and the right image capturing unit 1R, and processes this data to calculate the parallax. Parallax is the difference in image coordinates in which the same object is captured, which is caused by differences in the positions of multiple image capturing units. Parallax is large for close distances and small for long distances, and it is possible to calculate distance from the parallax. In addition, the stereo matching unit 2 calculates the parallax of the left image P L and right image P R For example, distortion of the image data is corrected so that objects at the same height and the same depth distance, known as a central projection or perspective projection model, are aligned horizontally in the image coordinates. The reason for the correction to align them horizontally is because the left imaging unit 1L and the right imaging unit 1R are arranged side by side in the left-right direction. The corrected left image P L and right image P R Using the above, one of these is used as reference image data that serves as a reference, and the other is used as comparison image data that serves as a comparison target, and the parallax is calculated.

[0026] To calculate the parallax, first align the vertical coordinates of the vanishing points of the reference image data and the comparison image data. Then, for each coordinate in the reference image data, determine which horizontal coordinates of the same vertical coordinate in the comparison image data represent the same object, using methods such as SSD (Sum of Squared Difference) or SAD (Sum of Absolute Difference). However, methods other than SSD and SAD are also acceptable. For example, methods such as FAST (Features from Accelerated Segment Test) and BRIEF (Binary Robust Independent Elementary Features), which extract corner feature points and check whether they are the same feature points, may be combined to match the same object.

[0027] The monocular detection unit 3 detects the left image P L Or right image P R The monocular detection unit 3 detects a specific three-dimensional object that appears in the left image P in which the detected object appears. Here, a specific three-dimensional object is an object that needs to be detected in order to realize appropriate AD control or ADAS control, specifically, a pedestrian, another vehicle, a bicycle, etc., around the vehicle. The detection target of the monocular detection unit 3 is a three-dimensional object within a certain range from the vehicle-mounted camera. The detection result by the monocular detection unit 3 includes the left image P in which the detected object appears. L Or right image P R For example, the detection result is stored as the vertical and horizontal image coordinates of the top left and bottom left of a rectangular frame (hereinafter referred to as the "monocular detection frame") that surrounds the detected object.

[0028] The stereo detection unit 9 detects, as a three-dimensional object, a location within a certain size range at the same distance from the parallax images created by the stereo matching unit 2. The detection result by the stereo detection unit 9 includes parallax image coordinate information of the detected object. For example, the detection result is stored as the vertical and horizontal image coordinates of the top left and bottom left of a rectangular frame (hereinafter referred to as the "stereo detection frame") that surrounds the detected object.

[0029] The type identification unit 13 identifies the type of object detected by the monocular detection unit 3 or the stereo detection unit 9 using an identification network such as a template image, a pattern, or a machine learning dictionary. If the type of object identified by the type identification unit 13 is a four-wheeled vehicle, a two-wheeled vehicle, or a pedestrian, it also identifies the direction of the front of the object, which is the direction of movement of the object, and identifies whether the object is a moving object (a three-dimensional object moving against the background) that is expected to cross in front of the vehicle. The direction of the front of these moving objects is the direction of the front of the object relative to the in-vehicle camera device 100. In this embodiment, operation is limited to movement directions that are close to perpendicular. Whether the movement direction is close to perpendicular is determined based on the angle of view and the direction of the front of the object relative to the in-vehicle camera device 100. For example, in the case of an object captured on the right side of the in-vehicle camera device 100, the face and side of the object are captured in the direction of travel. In the case of a 45° angle, if the front and side are viewed at a 45° angle, the object is determined to be moving in a direction that is close to perpendicular. The type identification unit 13 transmits the result of the determination as to whether the object is moving in a direction nearly perpendicular to the object to the speed calculation unit 10 .

[0030] The monocular distance measuring unit 4 identifies the position of a specific object detected by the monocular detection unit 3, and calculates the distance and direction from the left image capturing unit 1L or the right image capturing unit 1R. For example, the identified distance and direction are expressed in a coordinate system that can identify the position on a plane of the depth distance and the lateral distance in front of the vehicle. However, information expressed in a polar coordinate system that represents the Euclidean distance, which is the distance from the camera, and the direction may also be held, and the mutual conversion between the two axes of depth and lateral can be performed using trigonometric functions. In addition, the monocular distance measuring unit 4 calculates the distance and direction from the left image P L , Right image P R , and the composite image P C Using an overhead image projected onto the road surface, the position is identified from the vertical and horizontal coordinates of the overhead image and the vertical and horizontal scale of the overhead image relative to the actual road surface. However, using an overhead image to identify the position is not essential. The position may also be identified by performing geometric calculations using the external parameters of the camera position and orientation, the focal length, the pixel pitch of the image sensor, and information on the distortion of the optical system.

[0031] The stereo ranging unit 12 identifies the position of an object detected by the stereo detection unit 9 and determines its distance and direction. For example, the identified distance and direction are expressed in a coordinate system that can identify the position on a plane of the depth distance and lateral distance in front of the vehicle. However, information expressed in a polar coordinate system that represents the Euclidean distance, which is the distance from the camera, and the direction may also be held, and trigonometric functions may be used to convert between the two axes, depth and lateral. The stereo ranging unit 12 also calculates the depth distance from the parallax of the object. If there is variation in the calculated parallax of the object, the stereo ranging unit 12 uses the average or most frequent value. If the parallax variation is large, a method of identifying specific outliers may be used. The lateral distance is calculated using trigonometric functions from the horizontal angle of view of the detection frame of the stereo detection unit 9 and the depth distance.

[0032] The template creation unit 5 selects one of the captured images and cuts out a specific region from this captured image (hereinafter referred to as the "detection image") to create a template image. Specifically, the template creation unit 5 creates a template image for searching for similar locations from pixel information inside and around the monocular detection frame of the object detected by the monocular detection unit 3. This template image is enlarged or reduced to a predetermined image size. When enlarging or reducing the template image, the aspect ratio is maintained or not significantly changed. After the enlargement or reduction process, the pixel brightness values ​​themselves or the reduced image is divided into multiple small regions (hereinafter also referred to as "kernels") and the relationship between the brightness values ​​within the kernels is stored as the image features. Image features can be extracted in various ways, including the average brightness difference between the left and right or top and bottom within the kernel, the average brightness difference between the periphery and the center, and the average and variance of brightness, but any of these may be used in this embodiment.

[0033] In camera images, the background is reflected around the target, and when the image is cut out, the background is mixed in. However, the background changes depending on the time, even for the same target, as the target and camera move. In this invention, we want to find the same object at different times, so we use machine learning to determine the shape and texture of the target type to be tracked using camera images, and store which feature values ​​should be retained and with what weights to search for similar locations as a tracking network.

[0034] The image storage unit 6 stores the left image P L and right image P R The left image P L and right image P R The images are stored until a certain number of images are accumulated. A temporary storage device such as DRAM (Dynamic Random Access Memory) is used for this storage. The addresses and ranges within the storage device to store the images are determined in advance, and the destination address is changed in sequence for each captured image, and after one cycle, the area where the oldest image was stored is overwritten. Because the addresses are determined in advance, there is no need to notify the address when reading out the stored images.

[0035] The similar part search unit 7 searches for parts similar to the template image created by the template creation unit 5 from among the images stored in the image storage unit 6. Hereinafter, the image for which the similar part search unit 7 searches for parts similar to the template image will be referred to as the "search target image." The search target image is an image different from the image at the time of detection by the monocular detection unit 3. The image at which the time is selected is basically an image captured before or after the image at the time of detection. Similar parts are likely to exist near the monocular detection frame coordinates at the time of detection, and it is expected that there will be little change in brightness due to changes in the orientation of the detected object or changes in exposure compared to the template image, making high-precision search possible.

[0036] The target is tracked by determining the position of the same target at each time based on the search results, the detection results of the monocular detection unit 3, the detection results of the stereo detection unit 9, the distance measurement results of the monocular distance measurement unit 4, and the distance measurement results of the stereo distance measurement unit 12. This process of creating a template image from an image at a certain time and searching for and tracking similar locations in images taken at times different from that certain time is called image tracking processing using a tracking network.

[0037] However, if the image capture interval is short, an image from two or more previous times may be selected as the search target image. The image to be selected may also be changed depending on the vehicle speed. For example, if the vehicle speed is above a threshold, an older image taken at a time close to the captured time of the detected image may be selected, and if the vehicle speed is slower than the aforementioned threshold, an image even older than the aforementioned older image taken at a close time may be selected. If the vehicle speed is fast, an image taken at a time close to the detection time will not look significantly different from the template image, making it easier to ensure search accuracy. Furthermore, if the vehicle speed is slow, the position of the detected object in the image for which similar parts are searched will change significantly from the detection time, improving the accuracy of the calculated speed.

[0038] The angle of view determination unit 8 determines the horizontal angle of view of the camera of a specific object that appears in the similar location determined by the similar location search unit 7. However, if the height is determined as well as the position, the vertical image is also determined. Also, even if there is a possibility that the camera may roll, the speed can be calculated with high accuracy by determining both the horizontal and vertical angles of view. The horizontal angle of view can be found using a trigonometric function based on the ratio of the depth distance to the lateral distance.

[0039] The speed calculation unit 10 receives the distance measurement results from the monocular distance measurement unit 4 or the stereo distance measurement unit 12, the angle of view of the similar location from the angle of view identification unit 8, and vehicle behavior information from the vehicle information input unit 11, and calculates the speed of the detected specific object. However, instead of receiving the vehicle speed from the vehicle information input unit 11, the speed calculation unit 10 may use the differential value of the calculated relative depth distance as the vehicle speed. This is useful when the depth distance can be measured with high accuracy, but the lateral distance is measured with low accuracy. Furthermore, using the depth speed instead of the vehicle speed allows for accurate prediction of collisions with non-orthogonal crossing vehicles.

[0040] Vehicle information input unit 11 receives host vehicle speed information from vehicle speed sensor 31 that measures the speed of the host vehicle, host vehicle steering angle information from vehicle steering angle sensor 32 that measures the steering angle of the steering wheels, and host vehicle turning speed from yaw rate sensor 33 that measures the host vehicle turning speed. Note that vehicle information input unit 11 may be realized by a communication module compatible with a communication port (for example, IEEE802.3), or may be realized by an AD converter that can read voltages and currents.

[0041] The driver operation input unit 14 is connected to a steering sensor 34 that acquires information about the angle at which the steering wheel is turned, an accelerator pedal sensor 35 that acquires information about the amount of depression of the accelerator pedal, and a brake pedal sensor 36 that acquires information about the amount of depression of the brake pedal, and receives information about how much the driver operated the steering wheel, accelerator pedal, and brake pedal, and when they operated them.

[0042] The identification history storage unit 15 stores the three-dimensional objects (hereinafter referred to as "targets") identified by the type identification unit 13 together with the target identifier and the time of the target. The target identifier is a unique code assigned to each target, and for example, if one vehicle and three pedestrians are identified as targets, identifiers such as Vehicle A, Pedestrian A, Pedestrian B, and Pedestrian C are assigned to each target.

[0043] The tracking history storage unit 16 stores whether the template creation unit 5 and the similar part search unit 7 succeeded in creating a template image, the template image when searching for a similar part at each time, whether the search for a similar part was successful, and the image detected as a similar part, together with the target identifier and time.

[0044] The position history memory unit 17 stores the position of the same target at each time, along with the target's identifier and time, based on the search results, the detection results of the monocular detection unit 3, the detection results of the stereo detection unit 9, the distance measurement results of the monocular distance measurement unit 4, and the distance measurement results of the stereo distance measurement unit 12.

[0045] The speed history storage unit 18 stores the speed calculated by the speed calculation unit 10 together with the identifier of the target and the time.

[0046] The control feedback unit 23 is connected to the vehicle control system 200 and inputs control feedback information such as the vehicle control content and time controlled by the vehicle control system 200, the target identifier, the reason for the control decision, the content of the control cancellation, and the reason for the control cancellation.

[0047] The erroneous control determination unit 21 receives control feedback information of the vehicle from the control feedback unit 23 and driver operation information from the driver operation input unit 14. The erroneous control determination unit 21 also determines whether the vehicle control system 200 performed vehicle control based on the target information output from the in-vehicle camera device 100, initiated an alarm or automatic braking, and then canceled the control due to an operation by the driver. If such a driving operation has occurred, the erroneous control determination unit 21 determines that erroneous control has occurred by the vehicle control system 200. Details of the erroneous control determination unit 21 will be described later.

[0048] The error identification identifying unit 19 identifies cases of error identification among the identification results of targets that were the subject of erroneous control after the erroneous control is determined to be caused by the erroneous control determining unit 21. The details of the error identification identifying unit 19 will be described later.

[0049] The erroneous tracking identification unit 20 identifies erroneous tracking cases from among the tracking results of targets that were the subject of erroneous control after the erroneous control has been determined by the erroneous control determination unit 21. The erroneous tracking identification unit 20 will be described in detail later.

[0050] The erroneously recognized image storage unit 22 extracts and stores images before and after the erroneous identification or erroneous tracking identified by the erroneous identification identifying unit 19 or the erroneous tracking identifying unit 20.

[0051] The additional learning unit 24 additionally learns the AI ​​network for recognizing the external environment using images before and after the misidentification or mistracking, which are stored in the misrecognition image storage unit 22.

[0052] The process of generating learning data and the additional learning process using the generated learning data by the vehicle-mounted camera device 100 configured as described above will be described below with reference to specific examples.

[0053] <An example of how misidentification occurs, which is one type of misrecognition> 4 is a bird's-eye view showing an example of a situation in which an erroneous identification occurs as a result of the type identification unit 13 of the vehicle-mounted camera device 100 recognizing the external environment using an identification network, etc. Note that, although the sampling period of the external environment is set to 200 ms here, the sampling period is not limited to this example.

[0054] Figure 4(a) shows the time T 200 ms before time T0 (described later). -1 At this point, the vehicle is traveling at a constant speed on a straight road, and the vehicle-mounted camera device 100 attached to the vehicle has not detected any three-dimensional objects ahead of the vehicle.

[0055] FIG. 4(b) is an overhead view illustrating the external environment of the vehicle at time T0 when automatic control (automatic deceleration) is initiated. At this point, the vehicle-mounted camera device 100 has erroneously detected a non-existent pedestrian as a result of processing the captured image using a somewhat problematic identification network or the like. Therefore, the vehicle control system 200, which has received the detection result from the vehicle-mounted camera device 100, controls the braking system of the vehicle and rapidly decelerates the vehicle to prevent contact with the non-existent pedestrian. Note that a situation in which a non-existent pedestrian is erroneously detected is, for example, a situation in which a mass of exhaust gas is mistakenly identified as a pedestrian.

[0056] 4(c) is a bird's-eye view illustrating the external environment of the vehicle at time T1, 200 ms after time T0. At this point, the driver realizes that some abnormality (misidentification) has occurred in the in-vehicle camera device 100 because automatic deceleration has started even though there is no reason to brake the vehicle.

[0057] 4(d) is an overhead view illustrating the external environment of the vehicle at time T2, 400 ms after time T0. At this point, the driver confirms that the road ahead is safe (no vehicles ahead, no pedestrians around, etc.), and depresses the accelerator pedal to accelerate the vehicle to restore the decelerated speed to the speed before deceleration.

[0058] 4(e) is an overhead view illustrating the external environment of the host vehicle at time T3, 600 ms after time T0. At this point, the host vehicle has entered the area of ​​the pedestrian that was mistakenly identified at time T0, but since there is no pedestrian in that area, the host vehicle can pass through the area safely.

[0059] As is clear from the above, if the driver performs a manual driving operation contrary to the automatic control immediately after the vehicle control system 200 starts automatic control based on the identification result of the in-vehicle camera device 100, and no contact occurs with the identified three-dimensional object, it can be determined that an erroneous identification has occurred in the in-vehicle camera device 100. The device of the present invention stores and additionally learns the image of the erroneously identified pedestrian target at time T0, when it is determined that an erroneous identification has occurred.

[0060] <How to save misclassified images and how to use saved images for additional learning> Figure 5 is a flowchart that applies the mechanism explained in Figure 4, and shows a method for saving an image at the time when an incorrect classification occurs in the vehicle-mounted camera device 100, and then performing additional learning on the classification network, etc. based on the saved image. Each step will be explained in turn below.

[0061] First, in step S1, the erroneous control determination unit 21 determines whether or not automatic control for collision prevention has been activated based on data from the vehicle control system 200 obtained by the control feedback unit 23. Examples of automatic control for collision prevention include automatic braking for collision avoidance or mitigation, control for sounding a collision alarm, and steering control for obstacle avoidance. If automatic control based on the output of the in-vehicle camera device 100 has been activated, the process proceeds to step S2; if it has not been activated, step S1 is executed again after a certain period of time.

[0062] Next, in step S2, the erroneous control determination unit 21 determines whether or not a manual driving operation contrary to automatic control has been performed, based on data from the steering sensor 34, accelerator pedal 35, etc. obtained by the driver operation input unit 14. As illustrated in Fig. 4, if the driver performs a manual operation contrary to automatic control after automatic control to prevent contact with a detected three-dimensional object has been activated, it is considered that the driver performed the manual operation after confirming safety, and therefore it can be determined that the automatic control to avoid collision by the vehicle control system 200 was erroneous. Therefore, in this step, if a manual operation contrary to automatic control has been performed, the process proceeds to step S3, and if a manual operation contrary to automatic control has not been performed, the process returns to step S1.

[0063] A specific example of a situation in which it is determined in this step that the decision made by the cruise control system 200 is incorrect will be described.

[0064] <<First example of miscontrol>> Assume a situation where the vehicle control system 200 determines a collision with an object and starts an automatic brake aimed at stopping before the collision point. In this case, after the automatic brake is started and before stopping, if the driver steps on the accelerator pedal with a slightly strong depression amount (specifically, between a predetermined threshold Th1 and threshold Th2 (Th1 < Th2)), the operation is received as an operation intended to release the automatic brake, and the automatic brake control is released. As a result, as in the example of Fig. 4(e), the vehicle will pass before the collision point while accelerating. Thus, when the driver passes the collision point by operating the accelerator pedal after the vehicle control system 200 starts the automatic brake to stop before the collision point, it is determined that a driving operation contrary to the automatic control has been performed, and the process proceeds to step S3.

[0065] In this example, the operation is accepted as an operation to release the stop control only when the depression amount of the accelerator pedal is within the above-mentioned predetermined range, and this is due to the following reasons. That is, when the driver strongly steps on the accelerator pedal beyond the threshold Th2, there is a possibility that the driver has misunderstood the accelerator pedal as the brake pedal and stepped on it strongly, and it is considered inappropriate to accept this as an operation to release the stop control. Also, when the driver continues to step on the accelerator pedal with a weak force that does not reach the threshold Th1, there is a possibility that the driver has not noticed the possibility of a collision, and it is considered inappropriate to accept this as an operation to release the stop control. Therefore, the automatic brake is released only when the depression amount is between the threshold Th1 and the threshold Th2, and it is treated as a miscontrol.

[0066] <<Second example of miscontrol>> Assume a situation in which the vehicle control system 200 determines a collision with a target and initiates automatic braking with the goal of stopping the vehicle just before the collision point. In this case, after the vehicle is stopped by the automatic braking, the driver depresses the accelerator pedal to restart the vehicle and passes the collision point within a predetermined time. This time is, for example, 0.5 seconds, which is sufficiently shorter than the time it takes for an obstacle to move away if it is present at the collision point. The time it takes for the obstacle to move away is determined based on the crossing speed of the obstacle. If the collision point is at the center of the vehicle, it is the time it takes for the vehicle to move a vehicle width at the crossing speed.

[0067] If the target is a non-vehicle such as a pedestrian, the pedestrian may be surprised and stop in a situation where there is a possibility of a collision with the host vehicle, so the time for the pedestrian to stop for a few seconds may be added to the time for the vehicle to move. As described above, if the vehicle control system 200 stops the vehicle before the collision point, and the driver operates the accelerator pedal and the vehicle restarts and passes the collision point within the short time, the system determines that a driving operation contrary to automatic control has been performed, and proceeds to step S3.

[0068] <<Third example of miscontrol>> If the vehicle control system 200 determines that a collision with a target object will occur, but the distance to the target object is short and the vehicle cannot be stopped before the collision point even if automatic braking is initiated quickly, the vehicle control system 200 will apply automatic braking to mitigate the collision. If the driver depresses the accelerator pedal with a depression amount between threshold values ​​Th1 and Th2 after the automatic braking has been initiated, the vehicle control system 200 determines that the driver intends to release the automatic brake and releases the automatic brake control. If the driver operates the accelerator pedal to release the automatic brake under circumstances in which the vehicle cannot be stopped before the collision point, it is determined that a driving operation contrary to automatic control has been performed, and the system proceeds to step S3.

[0069] <<Fourth Example of Miscontrol>> In some cases, the vehicle control system 200 determines a collision with a target and controls a collision avoidance operation by automatically steering the vehicle to the right or left of the target to avoid the target. In this case, if the driver performs a steering operation that interferes with the automatic steering operation, it is determined that a driving operation contrary to automatic control has been performed, and the process proceeds to step S3. For example, the steering operation may occur when the vehicle control system 200 controls the steering wheel to turn 45 degrees to the right through the automatic steering operation, but the driver applies a force in the counterclockwise direction, resulting in a turn that is sufficiently smaller than 45 degrees, for example, only about 22 degrees.

[0070] In step S3, the error control determination unit 21 calculates the driving trajectory (trajectory of the vehicle's position) after automatic control based on the outputs of the vehicle speed sensor 31, vehicle steering angle sensor 32, and yaw rate sensor 33, and stores the calculated trajectory together with the driving operation. Specifically, as a result of the driver's driving operation, information on accelerator pedal operation, brake pedal operation, steering operation, vehicle speed, vehicle steering angle, and vehicle yaw rate is obtained from various sensors, and the driving trajectory required for the determination in the subsequent step S4 is calculated and estimated.

[0071] In step S4, assuming that the identification result at the start of automatic control was correct, the error identification identification unit 19 identifies the occurrence of an error by determining whether the vehicle was manually driven along a path that could avoid a collision with the identified object. Specifically, the error identification unit 19 compares the identification history and position history stored in the identification history storage unit 15 and the position history storage unit 17 with the path of the vehicle estimated in step S3, and determines whether the identification history includes collision avoidance with pedestrians, bicycles, vehicles, etc., for all historical positions of the identification results set by the vehicle control system 200, and stores the historical positions, times, and identifiers of the objects where collision avoidance was determined to be unavoidable. If there is even one historical position that is determined to be a path where collision avoidance was unavoidable, the unit determines that the driver selected that path after determining that it was safe to pass that position (i.e., there was an error in identification that caused the vehicle control system 200 to activate an automatic control that was not actually necessary), and proceeds to step S5. On the other hand, if not, it is determined that automatic control by the vehicle control system 200 was necessary (that is, the identification that caused the vehicle control system 200 to invoke automatic control was appropriate), and the process returns to step S1.

[0072] In step S5, the misidentification identifying unit 19 reads the misidentified image temporarily stored from the image storage unit 6 and stores it in the misrecognized image storage unit 22. Specifically, the identified image corresponding to the stored historical position, time, and target identifier is saved in memory. This makes it possible to limit the number of images saved in the misrecognized image storage unit 22 to, for example, at most a few images (a data volume of about several hundred KB) that have caused misidentification, thereby significantly reducing the storage capacity of the misrecognized image storage unit 22 compared to when all images captured during driving are saved as learning data.

[0073] In step S6, the additional learning unit 24 performs additional machine learning on the classification network, etc., using the misclassified images stored in the misrecognized image storage unit 22 in step S5 as training data. This reduces the frequency of additional machine learning and reduces the computational load, while improving the quality of the classification network, etc., through additional machine learning. In the additional learning in this step, the misclassified images stored in the misrecognized image storage unit 22 may be used as training data as is, or training data obtained by processing the stored misclassified images (e.g., training data obtained by enlarging or reducing the misclassified images, or training data obtained by rotating the misclassified images) may be used as training data. This makes it possible to learn situations other than images that perfectly match the coordinates at the time of misclassification, and to robustly suppress misclassification even when the detection frame is blurred.

[0074] When the identification image is saved in step S5, it is desirable to save a label specifying the type of misidentification together with the identification image. Since the vehicle control system 200 of this embodiment is designed to change the decision on whether to execute vehicle control depending on the type of detected target (another vehicle, pedestrian, bicycle, etc.), a mistaken decision on the type of target is considered to be a possible cause of the erroneous control. Therefore, the label attached to the identification image records an identifier specifying the identification network that made the mistake and the type of target that the identification network misidentified.

[0075] In order to teach a classification network that has made an erroneous classification that its classification was incorrect, type-class information on how the target was misclassified is also required, along with the training data (images). Therefore, by storing type-class information in the label in step S5, the image at the time of the misclassification can be used in step S6 to pinpoint and perform additional machine learning on the problematic classification network, etc., identified by the label, thereby improving the problematic classification network, etc. Furthermore, since non-problematic classification networks, etc. can be excluded from the target of additional learning, it is possible to avoid adverse effects such as erroneous learning of normal classification networks, etc., and an increase in the computational load caused by performing unnecessary additional learning on normal classification networks, etc.

[0076] <An example of how mistracking occurs, which is one type of misrecognition> FIG. 6A is a bird's-eye view showing an example of a situation in which erroneous tracking occurs as a result of the similar part searching unit 7 of the vehicle-mounted camera device 100 recognizing the external environment using a template image or a tracking network with some problem, and FIG. 6B is a legend for FIG. 6A. Note that the bird's-eye view shown in the figure is, for example, a captured left image P L and the right image P R The bird's-eye view map is created by taking into consideration the time-dependent changes in the projected image that has been affine transformed using a predetermined affine table, and the time-dependent changes in the position of the vehicle itself. However, since the method of creating an overhead view map of the area around the vehicle using this method is a well-known technology, detailed explanations will be omitted.

[0077] 6A(a) to (c) show the time T 600 ms, 400 ms, and 200 ms before time T0, which will be described later. -3 , T -2 , T -1 1 is a bird's-eye view illustrating an example of the external environment of the vehicle at time T. At these times, the vehicle is traveling at a constant speed on a straight road, and the vehicle-mounted camera device 100 is tracking the movement of a pedestrian walking on the right sidewalk using a tracking network or the like. -3 ~T -1During this period, the vehicle control system 200 determines that the host vehicle will not come into contact with a pedestrian, and therefore does not perform automatic control to avoid contact with a pedestrian at this time.

[0078] FIG. 6A(d) is an overhead view illustrating the external environment of the vehicle at time T0 when automatic control (automatic deceleration) is initiated. At this point, the vehicle-mounted camera device 100 processes the captured image using a somewhat problematic template image, tracking network, etc., resulting in the erroneous tracking of a pedestrian who is actually on the sidewalk as having run out onto the roadway. Therefore, the vehicle control system 200, which receives the tracking results from the vehicle-mounted camera device 100, controls the vehicle's braking system to rapidly decelerate the vehicle to prevent contact with the non-existent pedestrian on the road. An example of a situation in which a non-existent pedestrian on the road may be a situation in which a rising flag or vegetation in the pedestrian's path sways due to a sudden gust of wind, causing the swaying to be mistaken for a pedestrian running out onto the roadway.

[0079] Here, the transition of the position of the target (pedestrian) detected by the vehicle-mounted camera device 100 and the method of determining whether to start contact avoidance control with the detected target (pedestrian) will be described with reference to FIG. 6B. -3 , O.B. -2 , O.B. -1 , OB0 are at time T -3 , T -2 , T -1 , T0 indicates the position of the detected target (pedestrian). Note that the position OB0 is a false detection. Also, v -1 , v0, v1 are the time T -3 ~T -2 , time T -2 ~T -1 , time T -1 The movement vector of the target (pedestrian) during the period from T0 to T1 is obtained by converting the velocity vector of the target (pedestrian) calculated by the vehicle-mounted camera device 100 into the amount of movement for each period. L , E R are the left and right ends of the travel range of the host vehicle, and indicate the left and right ends of the range in which the host vehicle is predicted to travel based on the steering angle, yaw rate, and speed of the host vehicle.

[0080] In this embodiment, when a moving target enters a predetermined contact determination range that includes the host vehicle's travel range, the vehicle control system 200 determines that there is a possibility of contact with the target and initiates automatic control, such as braking or steering, to avoid contact with the target. Therefore, in FIG. 6B , at time T0 when the target position OB0 within the contact determination range is detected, the vehicle control system 200 initiates automatic control, such as sudden deceleration of the host vehicle. Note that the length of the host vehicle's travel range is limited to the range within which the host vehicle is predicted to travel within a predetermined period (e.g., within 5 seconds). Even if a target is present, distant areas where immediate contact avoidance control is not required do not need to be included in the host vehicle's travel range at that time. The width of the contact determination range may be varied according to road type information obtained from a global navigation satellite system (GNSS) or the like. For example, the width may be narrower on expressways where the possibility of sideways approaching targets (e.g., pedestrians or other vehicles) is low, and wider on general roads where the possibility of sideways approaching targets is high.

[0081] 6A(e) is an overhead view illustrating the external environment of the host vehicle at time T1, 200 ms after time T0. At this point, the driver realizes that some abnormality (mistracking) has occurred in the in-vehicle camera device 100 because automatic deceleration has begun even though there is no reason to brake the host vehicle. Therefore, once the driver confirms that the road ahead is safe (there are no vehicles ahead, no pedestrians around, etc.), the driver depresses the accelerator pedal to accelerate the host vehicle, restoring the decelerated speed to the speed before deceleration. As a result, the host vehicle approaches the area of ​​the pedestrian that was falsely detected, but because there is actually no pedestrian in that area (the pedestrian is actually on the right sidewalk), the host vehicle is able to pass through the area safely.

[0082] As is clear from the above, if the vehicle control system 200 starts automatic control based on the tracking results of the in-vehicle camera device 100, and the driver performs a manual driving operation that contradicts the automatic control, and no contact occurs with the tracked three-dimensional object, it can be determined that erroneous tracking has occurred in the in-vehicle camera device 100.

[0083] <Method for saving images before and after misclassification, and additional learning method using saved images> Figure 7 is a flowchart that applies the mechanism described in Figure 6A, and shows a method for saving images taken around the time when erroneous tracking occurred in the vehicle-mounted camera device 100, and then additionally learning template images, a tracking network, and the like based on the saved images. Each step will be explained in order below. Note that the processes in Figures 5 and 7 can be performed in parallel, and in the following, overlapping explanations of points common to the flowchart in Figure 5 will be omitted as necessary.

[0084] First, in step S1, the erroneous control determination unit 21 determines, based on data obtained by the control feedback unit 23 from the vehicle control system 200, whether or not automatic control for preventing collision has been activated.

[0085] Next, in step S2, erroneous control determination unit 21 determines whether or not a manual driving operation contrary to automatic control has been performed, based on data from steering sensor 34, accelerator pedal 35, etc. obtained by driver operation input unit 14. As described in Fig. 6A, if the driver performs a manual operation contrary to automatic control after automatic control for preventing contact with the target being tracked has been activated, it is considered that the driver performed the operation after confirming safety, and therefore it can be determined that the automatic control for preventing contact by vehicle control system 200 was erroneous.

[0086] In step S3, the error control determination unit 21 calculates the driving trajectory (trajectory of the vehicle's position) after automatic control based on the outputs of the vehicle speed sensor 31, vehicle steering angle sensor 32, yaw rate sensor 33, etc., and stores it together with the driving operation.

[0087] In step S4a, assuming that the tracking result at the start of automatic control was correct, the erroneous tracking identification unit 20 identifies the occurrence of erroneous tracking by determining whether the vehicle was driven manually along a path that would avoid contact with the tracked object.

[0088] In step S5a, the erroneous tracking identifying unit 20 reads from the image storage unit 6 images before and after the occurrence of erroneous tracking (for example, images taken one second before and after the automatic control start time) and stores them in the erroneously recognized image saving unit 22.

[0089] In step S6a, the additional learning unit 24 performs additional machine learning on the template image, tracking network, etc., using the images before and after the occurrence of mistracking, which were stored in the misrecognition image storage unit 22 in step S5a, as learning data. This makes it possible to suppress the frequency of additional machine learning and reduce the computational load, while improving the quality of the template image, tracking network, etc., through the additional machine learning.

[0090] In step S5a of Fig. 7, it is also desirable to store a label that specifies the mode of mistracking together with the identification image, following the example of step S5 of Fig. 5. This allows additional learning in step S6a to be performed only on template images, tracking networks, etc. that require improvement, due to the same effect as described in Fig. 5, thereby obtaining various effects similar to those of Fig. 5.

[0091] It should be noted that, near the location identified in step S4a as the occurrence of mistracking, there is a possibility that an object identical to the template or the same object that appears in the image before the mistracking location may be present, and therefore, by tracking and expanding the area with a detection frame in the images before and after the mistracking location and storing the images, it is possible to store images that include the location where the same object appears, and it is possible to appropriately perform additional learning of the location that should be correctly tracked during additional learning.

[0092] FIG. 8 is an overhead view illustrating the behavior of the vehicle when an erroneous identification occurs and a method for determining whether the erroneous control or erroneous identification has occurred, taking into account the movement of a pedestrian, for a target that was identified at time T0 and triggered control of the vehicle. The pedestrian prediction range in FIG. 8(c) shows the predicted range in which a pedestrian may exist, calculated by multiplying the time elapsed since the pedestrian was identified in FIG. 8(b) by the pedestrian's movement speed. The pedestrian's movement speed may be set to a speed that is likely to be a pedestrian, assuming an average walking speed, for example, 5 km / h. The pedestrian presence radius in the legend in FIG. 8 indicates the radius of the pedestrian prediction range, and is calculated by multiplying the time elapsed since the pedestrian was identified, T, by the pedestrian's movement speed, V. P The length L is the product of

[0093] In the example of Fig. 8, the target identified at time T0 is determined to have been misidentified at time T3 because the vehicle itself covers the predicted pedestrian presence range. In the present invention, for the detected and identified targets at each time when a collision is judged during automatic control, the target at the time of misidentification is identified based on the degree of overlap between the predicted pedestrian range and the vehicle itself range.

[0094] FIG. 9 is a flowchart explaining the process of saving images before and after misidentification, taking into account the movement of pedestrians. Note that the overlapping explanation of points common to FIG. 5 is omitted. Step S7 in FIG. 9 is the process carried out after step S3, and is a process of estimating the predicted range of pedestrians after automatic control. The pedestrian presence radius is calculated by multiplying the elapsed time T after the pedestrian is identified and the moving speed V of the pedestrian. P The pedestrian presence range is determined by calculating the distance between the vehicle and the pedestrian.

[0095] The subsequent step S4b is a process of determining whether a manual operation to avoid an identified target has been performed, and determines whether a manual operation to avoid an identified target has been performed based on the travel trajectory verified in step S3 and the predicted pedestrian range estimated in step S7. If the travel trajectory of the host vehicle shows that the vehicle covers the predicted pedestrian range, it is determined that a manual operation to avoid the pedestrian's identified target has not been performed.

[0096] In this way, by taking into account the movement of pedestrians, the risk of misjudging an erroneous classification due to pedestrian movement can be reduced, and the effectiveness of additional learning of erroneous classification locations can be increased.

[0097] <Effects of this Example> According to the present embodiment described above, it is possible to automatically extract and save images taken before and after a misrecognition of the external environment from various images captured in a real driving environment, which will contribute to improving a problematic AI network. As a result, the in-vehicle camera device of this embodiment can use the automatically extracted and saved images as learning data without external cooperation, thereby allowing the problematic AI network to perform additional learning and improve the quality of the AI ​​network.

[0098] The present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0099] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as a semiconductor memory card.

[0100] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected.

[0101] In each of the above-described embodiments and modifications, the functional block configurations are merely examples. Some functional configurations shown as separate functional blocks may be configured as an integrated unit, or a configuration shown in a single functional block diagram may be divided into two or more functions. Furthermore, some of the functions of each functional block may be provided by other functional blocks.

[0102] The above-described embodiments and modifications may be combined with each other. Although various embodiments and modifications have been described above, the present invention is not limited to these. Other embodiments conceivable within the scope of the technical concept of the present invention are also included within the scope of the present invention. [Explanation of symbols]

[0103] 100...In-vehicle camera device 1L...Left imaging section 1R...Right imaging unit 2...Stereo matching section 3...Monocular detection unit 4... Monocular distance measuring section 5. Template Creation Department 6...Image storage section 7...Similar parts search section 8...Angle of view specification section 9...Stereo detection unit 10…Speed ​​calculation section 11...Vehicle information input section 12...Stereo distance measurement unit 13...Type identification section 14...Driver operation input section 15...Identification history memory unit 16...Tracking history memory section 17...Location history memory section 18...Speed ​​history memory section 19...Misidentification Identification Department 20...False Tracking Identification Department 21...Error control identification unit 22…Misrecognized image storage section 23...Control feedback section 24…Additional Learning Section 200...Vehicle control system

Claims

1. An in-vehicle camera device for identifying an object, a control feedback unit that receives feedback on automatic control of the host vehicle; an erroneous control determination unit that determines erroneous automatic control based on driving operation information of a driver; an image storage unit for storing images; The image captured under the actual driving environment at the time when the misidentification that was determined to be erroneous automatic control occurred is saved. Further, a misidentification identifying unit is provided that identifies that the erroneous automatic control is caused by a misidentification of the type of object, The image of the object located in the area in which the vehicle travels after the erroneous automatic control is saved, An in-vehicle camera device stores, as a misidentified image, an identified object whose predicted pedestrian range is covered by a vehicle on the vehicle's travel path.

2. An in-vehicle camera device for identifying an object, a control feedback unit that receives feedback on automatic control of the host vehicle; an erroneous control determination unit that determines erroneous automatic control based on driving operation information of a driver; an image storage unit for storing images; The image captured under the actual driving environment at the time when the misidentification that was determined to be erroneous automatic control occurred is saved. Further, a mistracking identification unit is provided that identifies that the erroneous automatic control is caused by mistracking of an object, An in-vehicle camera device characterized by storing images of objects located in an area in which the vehicle travels after erroneous automatic control.

3. The vehicle-mounted camera device according to claim 2, The vehicle-mounted camera device is characterized in that the image is saved with a label indicating mistracking.

4. The vehicle-mounted camera device according to claim 2, Further, a template creating unit is provided to create a template image to be used when tracking an object, An in-vehicle camera device characterized in that it stores both an image of an object trajectory determined to be erroneously tracked and a template image used to track the object trajectory.

5. An in-vehicle camera device for identifying an object, a control feedback unit that receives feedback on automatic control of the host vehicle; an erroneous control determination unit that determines erroneous automatic control based on driving operation information of a driver; an image storage unit for storing images; The image captured under the actual driving environment at the time when the misidentification that was determined to be erroneous automatic control occurred is saved. It has multiple networks that recognize the external environment of the vehicle, An in-vehicle camera device characterized in that the images stored in the image storage unit are labeled to indicate which network's machine learning the images are used for.

6. An image storage method for storing images used in training a network for identifying objects, comprising: a control feedback process for receiving feedback on the automatic control of the host vehicle; an erroneous control determination process for determining erroneous automatic control based on driving operation information of the driver; an image storage process for storing an image captured in an actual driving environment at the time when the misidentification that was determined to be erroneous automatic control occurred; a misidentification identification process for identifying that the erroneous automatic control is caused by a misidentification of an object type; The image of the object located in the area in which the vehicle travels after the erroneous automatic control is saved, An image storage method for storing, as a misidentified image, an identified object whose predicted pedestrian range is covered by a vehicle on the vehicle's travel path.

7. An image storage method for storing images used in training a network for identifying objects, comprising: a control feedback process for receiving feedback on the automatic control of the host vehicle; an erroneous control determination process for determining erroneous automatic control based on driving operation information of the driver; an image storage process for storing an image captured in an actual driving environment at the time when the misidentification that was determined to be erroneous automatic control occurred; a mistracking identification process for identifying that the erroneous automatic control is caused by erroneous tracking of an object; An image saving method characterized by saving an image of an object located in an area in which a vehicle travels after erroneous automatic control.

8. The image storage method according to claim 7, An image saving method, characterized in that the image is saved with a label indicating mistracking attached thereto.

9. The image storage method according to claim 7, Furthermore, a template creation process is provided to create a template image to be used when tracking an object, An image saving method characterized by saving both an image of an object trajectory determined to be mistracked and a template image used to track the object trajectory.

10. An image storage method for storing images used in training a network for identifying objects, comprising: a control feedback process for receiving feedback on the automatic control of the host vehicle; an erroneous control determination process for determining erroneous automatic control based on driving operation information of the driver; An image storage process is provided to store images captured in an actual driving environment at the time when an erroneous identification that was determined to be erroneous automatic control occurred, An image saving method characterized in that the images saved in the image saving process are labeled with a label indicating which of multiple networks that recognize the external environment of the vehicle the image is used for machine learning.

Images