Image recognition device and stored image determination method

Abstract

An image recognition device that performs object detection on an input image using an image recognition program includes: a storage unit that stores a current program, which is an existing image recognition program, and a new program, which is a new image recognition program; a difference extraction unit that extracts a detection difference, which is the difference between the object detection result of the current program and the image recognition result of the new program for the same input image; a transmission determination unit that determines, on the basis of the occurrence of the detection difference, whether or not the input image needs to be stored; and a storage unit that outputs the input image determined to be stored by the transmission determination unit to the outside of the image recognition device, or stores the input image determined to be stored by the transmission determination unit in the image recognition device.

Description

Technical Field

[0001] The present invention relates to an image recognition device and a method for determining an image to be saved. Background Art

[0002] In an application program in which a serious accident related to human life may occur due to a program defect, it is required to thoroughly perform performance verification before actually using the program. For example, in an automotive program, after verification using a large number of test patterns in a virtual development environment such as a simulator, a public road test drive of hundreds of thousands of kilometers is performed in a test vehicle to confirm no defects and ensure safety. The same verification using a test vehicle is required when the program is updated, but there is a problem in that a large amount of time and human cost for verifying the results are required in the verification using an actual machine that originally works in an actual environment. Patent Document 1 discloses an operation verification device: The operation verification device includes: a division unit that divides each of the plurality of control processes included in a first program and a plurality of control processes including at least a part of the plurality of control processes whose control processes have been changed, into a parallel process that realizes parallel processing of the plurality of control processes and a function sequential process that realizes the functions of the plurality of control processes, outputs the first program in which each of the plurality of control processes is divided into the parallel process and the function sequential process as a first divided completion program, and outputs the second program in which each of the plurality of control processes is divided into the parallel process and the function sequential process as a second divided completion program; and a cause presumption unit that, when a function defect is detected as a function defect for the second divided completion program, presumes the function sequential process different between the first divided completion program and the second divided completion program as the cause of the function defect, and when a defect caused by parallel processing is detected as a parallel defect for the second divided completion program, presumes the parallel process different between the first divided completion program and the second divided completion program as the cause of the parallel defect. Prior Art Documents Patent Documents

[0003] Patent Document 1: International Publication No. 2018 / 150504 Summary of the Invention Problems to be Solved by the Invention

[0004] In the invention described in Patent Document 1, an image for evaluating a newly generated program cannot be appropriately determined. Technical Means for Solving the Problems

[0005] The image recognition device according to the first aspect of the present invention is an image recognition device that performs object detection on an input image using an image recognition program. The image recognition device includes: a storage unit that stores a current program as the existing image recognition program and a new program as the new image recognition program; a difference extraction unit that extracts a difference, i.e., a detection difference, between the object detection result of the current program and the image recognition result of the new program for the same input image; a transmission determination unit that determines whether to save the input image based on the occurrence status of the detection difference; and a storage unit that outputs the input image determined to be saved by the transmission determination unit to the outside of the image recognition device, or saves the input image determined to be saved by the transmission determination unit in the image recognition device. The saved image determination method according to the second aspect of the present invention is a saved image determination method executed by an image recognition device that performs object detection on an input image using an image recognition program. The image recognition device includes a storage unit that stores a current program as the existing image recognition program and a new program as the new image recognition program. The saved image determination method includes: a difference extraction step of extracting a difference, i.e., a detection difference, between the object detection result of the current program and the image recognition result of the new program for the same input image; a transmission determination step of determining whether to save the input image based on the occurrence status of the detection difference; and a storage step of outputting the input image determined to be saved by the transmission determination step to the outside of the image recognition device, or saving the input image determined to be saved by the transmission determination step in the image recognition device. Effects of the Invention

[0006] According to the present invention, it is possible to appropriately determine an image for evaluating a newly generated program. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 It is a configuration diagram of a vehicle equipped with the image recognition device according to the first embodiment. Figure 2 It is a configuration diagram of the image recognition device according to the first embodiment. Figure 3 It is a hardware configuration diagram of the image recognition device. Figure 4 It is a schematic diagram showing the processing of the difference extraction unit. Figure 5 It is a diagram showing a first example of the transmission determination unit. Figure 6 It is a diagram showing an example of the detection difference input to the time series filter. Figure 7 It is a diagram showing another example of the detection difference input to the time series filter. Figure 8 It is a diagram showing a second example of the transmission determination unit. Figure 9 It is a diagram showing an example of a risk determination diagram for reference of a spatial filter. Figure 10 It is a diagram showing a third example of the transmission determination unit. Figure 11 It is a configuration diagram of the image recognition device of Modification 1. Figure 12 It is a configuration diagram of the image recognition device of the second embodiment. Figure 13 It is a configuration diagram of the transmission determination unit in the second embodiment. Figure 14 It is a configuration diagram of a vehicle equipped with the image recognition device of the third embodiment. Figure 15 It is a diagram showing the relationship between the image recognition device, the control program, the second control program, and the control difference extraction unit. Detailed Embodiments

[0008] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Each embodiment is an example for explaining the present invention, and appropriate omissions and simplifications are made for clear explanation. The present invention can also be implemented in various other ways. Unless otherwise specified, each component can be singular or plural.

[0009] For easy understanding of the invention, the positions, sizes, shapes, ranges, etc. of the components shown in the drawings sometimes do not represent the actual positions, sizes, shapes, ranges, etc. Therefore, the present invention is not necessarily limited to the positions, sizes, shapes, ranges, etc. disclosed in the drawings. When there are multiple components having the same or similar functions, sometimes different additional symbols are attached to the same symbol for explanation. In addition, when it is not necessary to distinguish these multiple components, sometimes the additional symbols are omitted for explanation.

[0010] In each embodiment, there are cases where the processing performed for executing a program is described. Here, a computer executes a program through a processor (such as a CPU or GPU), while using storage resources (such as a memory), interface devices (such as communication ports), etc., and performs the processing determined by the program. Therefore, the entity that performs the processing for executing the program can also be regarded as the processor. Similarly, the entity that performs the processing for executing the program can also be a controller, device, system, computer, or node having a processor. The entity that performs the processing for executing the program only needs to be an arithmetic unit, and can also include a dedicated circuit that performs specific processing. Here, the dedicated circuit is, for example, an FPGA (Field Programmable Gate Array), an ASIC (Application Specific Integrated Circuit), a CPLD (Complex Programmable Logic Device), etc.

[0011] The program can be installed in a computer from a program source. The program source can be, for example, a program distribution server or a computer-readable storage medium. In the case where the program source is a program distribution server, the program distribution server can also include a processor and a storage resource for storing the program to be distributed, and the processor of the program distribution server distributes the program to be distributed to other computers. Additionally, in an embodiment, two or more programs can be implemented as one program, or one program can be implemented as two or more programs.

[0012] In recent years, in order to achieve highly automated driving, a surrounding recognition application program that uses an image recognition program applying a deep neural network (DNN: Deep Neural Network), which is one of machine learning, is becoming popular. It is known that compared with a rule-based algorithm, the performance change of a program using machine learning is more difficult to occur when the program is updated.

[0013] Therefore, for example, in a surrounding recognition application program, for a certain image, the object can be correctly recognized in the current version of the program (hereinafter referred to as the "current program"), but in the updated version of the program (hereinafter referred to as the "new program"), the recognition performance of the object may deteriorate. The deterioration of the recognition performance means that the object is recognized with a position offset, or is not detected because the reliability score used in the determination for recognition is lower than the threshold, etc.

[0014] Such a phenomenon occurs in a complex manner related to various parameters, in addition to features such as the size, color, orientation, and shape of the objects appearing in the image, but also related to the positional relationship of surrounding objects and the background, as well as noise caused by hardware incorporated during image photography. Therefore, since the detection results fluctuate each time image recognition processing is performed, when verifying the equivalence of the processing results of the current program and the new program, differences will occur between the two programs at a very high frequency.

[0015] For the purpose of ex post facto verification and program development, it is useful to save images with different processing results between the current program and the new program. However, if images are saved only because of differences in processing results, the number of saved objects will become huge. That is, if images with different recognition performances are saved in the current program and the new program, the number of saved objects will become extremely large. Hereinafter, a method for determining the images to be saved will be described.

[0016] In the following embodiments, an example of applying the present invention to an in-vehicle ECU for vehicle control, such as an Advanced Driver Assistance System (ADAS) and Autonomous Driving (AD), will be described. However, the present invention is not limited to in-vehicle ECUs for ADAS and AD. In addition, it can be used for update verification of Automatic Guided Vehicles (AGVs) and peripheral recognition AIs for construction machinery, and can also be used for update verification of AIs such as surveillance cameras, and can be applied to update verification of all information processing algorithms using machine learning such as image processing.

[0017] - First Embodiment - Hereinafter, with reference to Figures 1 to 10 a first embodiment of an image recognition device and a method for determining saved images will be described.

[0018] Figure 1It is a configuration diagram of a vehicle 9 equipped with an image recognition device 1. The vehicle 9 includes an image recognition device 1, a camera 91, an in-vehicle communication device 92, and a control program 93. The image recognition device 1 can communicate with the camera 91, the in-vehicle communication device 92, and the control program 93 using a known communication method. This communication can be wired or wireless. This communication includes, for example, IEEE802.3, Controller Area Network, IEEE802.11, etc. The camera 91 captures the surroundings of the vehicle 9 and outputs the captured image (hereinafter referred to as "captured image", "input image") to the image recognition device 1. The image recognition device 1 performs object detection on the captured image using an image recognition program and outputs the detection result to the control program 93. In addition, as described later, the image recognition device 1 outputs a part of the captured image to the in-vehicle communication device 92.

[0019] The in-vehicle communication device 92 wirelessly transmits the captured image output by the image recognition device 1 to an image storage server 99 existing outside the vehicle 9. The in-vehicle communication device 92 and the image storage server 99 can communicate directly wirelessly, can communicate via a communication relay point fixed on the ground such as a base station between the two, or can communicate via other vehicles. The image storage server 99 stores the captured image received from the in-vehicle communication device 92 in a non-volatile storage device (not shown).

[0020] The control program 93 performs calculations using the detection result output by the image recognition device 1. The content of the calculation executed by the control program 93 is arbitrary. For example, it can control the vehicle 9 according to the detection result, or can notify the driver of the vehicle 9 of the presence of an obstacle according to the calculation result.

[0021] Figure 2 It is a configuration diagram of the image recognition device 1. The image recognition device 1 includes a current program 11, a new program 12, a difference extraction unit 13, a transmission determination unit 14, and a trigger reception unit 19. The camera 91 outputs the captured image 31 to the current program 11, the new program 12, and the trigger reception unit 19. The current program 11 takes at least the captured image 31 as input and outputs a current detection result 32. The new program 12 takes at least the captured image 31 as input and outputs a new detection result 33. The difference extraction unit 13 takes the current detection result 32 and the new detection result as input and outputs a detection difference 34. The transmission determination unit 14 takes the detection difference 34 as input and outputs a transmission trigger 35. The trigger reception unit 19 takes the captured image 31 and the transmission trigger 35 as input and outputs the captured image 31. Since the trigger reception unit 19 outputs the captured image 31 for storage to the outside, it can also be called a "storage unit".

[0022] Both the current program 11 and the new program 12 are image recognition programs for inputting a captured image 31, and both detect objects. However, the stability of the operation of the current program 11 has been confirmed, while the operation of the new program 12 is in a state where it has not been fully confirmed. The new program 12 is, for example, a program that corrects omissions in detection and detection errors in the current program 11, or a program that is corrected in such a way as to be able to recognize new types of objects. The details of the operations of the current program 11 and the new program 12 are not particularly limited, and they can be programs based on DNN (deep neural network) using machine learning, or logic-based programs applying pattern recognition technology or the like.

[0023] The current program 11 and the new program 12 can each take various forms. For example, a single binary file, a combination of a binary file and a configuration file, a combination of a binary file and a library, etc. In addition, at least one of the current program 11 and the new program 12 can be a rewritable logic circuit.

[0024] The current detection result 32 output by the current program 11 and the new detection result 33 output by the new program 12 are the coordinates of the object in the captured image 31, the type of the object, and the detection reliability. The coordinates of the object are, for example, coordinates in an orthogonal coordinate system with the upper left of the captured image as the origin. The type of the object is, for example, a car, a pedestrian, a bicycle, etc. The current detection result 32 is output to the control program 93 for its original use, and is also output to the difference extraction unit 13 for comparison.

[0025] Since the new detection result 33 output by the new program 12 is the output result of the new program 12 that has not been fully verified, it is not used as the input for the post-processing of image recognition in the in-vehicle ECU, but is always used as verification information.

[0026] The difference extraction unit 13 extracts the difference between the current detection result 32 and the new detection result 33, and outputs it as a detection difference 34. The detailed processing of the difference extraction unit 13 will be described later. The transmission determination unit 14 determines whether the captured image 31 should be saved based on the occurrence status of the detection difference 34, and outputs a transmission trigger 35. Here, the occurrence status of the detection difference 34, for example, is the occurrence frequency, occurrence continuity, position fluctuation, etc. of the detection difference 34 in the time series. In addition, it can also be determined whether the position where the detection difference 34 occurs is within a predetermined area, and it can also be determined based on a combination of the above-mentioned determination conditions such as occurrence frequency and occurrence continuity and the determination condition of the difference occurrence position. The transmission trigger 35 can be a signal output only when the transmission determination unit 14 determines that it should be saved, or a signal that always outputs including data indicating whether saving is required. The detailed processing of the transmission determination unit 14 will be described later.

[0027] The trigger reception unit 19 outputs the captured image 31 to the vehicle exterior communication device 92 based on the transmission trigger 35. When the transmission determination unit 14 outputs the transmission trigger 35 only when it is determined that saving is required, the trigger reception unit 19 outputs the captured image 31 when the transmission trigger 35 is input. When the transmission determination unit 14 outputs the transmission trigger 35 including data indicating whether saving is required, the trigger reception unit 19 outputs the captured image 31 when the transmission trigger 35 includes data indicating that saving is required.

[0028] Figure 3 It is a hardware configuration diagram of the image recognition device 1. The image recognition device 1 is an electronic control device, i.e., an ECU, that includes a CPU 41 as a central processing unit, a ROM 42 as a read-only storage device, a RAM 43 as a readable and writable storage device, and an in-vehicle communication device 44. The CPU 41 loads and executes the program stored in the ROM 42 in the RAM 43, thereby performing the above various operations.

[0029] Instead of the combination of the GPU 40, CPU 41, ROM 42, and RAM 43, the image recognition device 1 can be implemented by an FPGA as a rewritable logic circuit and an ASIC (Application Specific Integrated Circuit) for specific purposes. In addition, the image recognition device 1 can also be implemented by a combination of different configurations instead of the combination of the GPU 40, CPU 41, ROM 42, and RAM 43, such as a combination of the CPU 41, ROM 42, RAM 43, and FPGA. Furthermore, a dedicated circuit (such as an artificial intelligence accelerator) for efficiently executing DNN can be mounted instead of the GPU 40. The in-vehicle communication device 44 corresponds to IEEE802.3 or Controller Area Network and realizes communication with the camera 91 and the vehicle exterior communication device 92.

[0030] In Figure 3 only one GPU 40, CPU 41, and RAM 43 are described, but two can be mounted respectively, and the current program 11 and the new program 12 can be executed by different hardware resources. In addition, the image recognition device 1 can be composed of multiple ECUs, and the current program 11 and the new program 12 can be executed by different ECUs.

[0031] Figure 4 It is a schematic diagram showing the processing of the differential extraction unit 13. In Figure 4In this, the processing result of the captured image 31 shown in the upper part is described. In this captured image 31, other vehicles traveling in front of the vehicle 9 are captured in the upper left, and the road surface is captured outside the upper left. The current program 11 and the new program 12 process this captured image 31 and respectively output the current detection result 32 and the new detection result 33. In the current detection result 32, the area of the vehicle is detected as shown by the dotted line, and in the new detection result 33, the road surface is detected. Hereinafter, the outer edge of the area detected by the current detection result 32 and the new detection result 33 is referred to as the "detection frame".

[0032] The difference extraction unit 13 extracts the differences in the coordinates of the detected objects, the types of objects, and the detection reliability between the current detection result 32 and the new detection result 33 as differences. Whether the coordinates are consistent can be determined, for example, by whether the IoU (Intersect of Union) for the detection frame is a specified threshold, for example, whether it is 0.5 or less. Specifically, when the area of the detection frame in the current detection result 32 is set as A32, the area of the detection frame in the new detection result 33 is set as A33, and the overlapping area of the two is set as B, the difference extraction unit 13 determines that the coordinates are consistent when the following formula 1 holds.

[0033] IoU = B / (A32 + A33 - B) > 0.5…(Formula 1)

[0034] In Figure 4 the example shown, since the detection frames of the current detection result 32 and the new detection result 33 do not overlap at all, B in formula 1 is zero, so the condition of formula 1 is not satisfied, and the difference extraction unit 13 determines that the coordinates are inconsistent. Then, as shown in the lower part of Figure 4 , the difference extraction unit 13 outputs the area of each inconsistent detection frame as the detection difference 34.

[0035] Figure 5 is a diagram showing the first example of the transmission determination unit 14. Figure 5 The shown transmission determination unit 14 includes a time series filter 141 and a trigger determination unit 144. The time series filter 141 takes the detection difference 34 as input and outputs the time series filter output 36. The time series filter 141 observes the occurrence status of the detection difference 34 in time series and quantifies the degree of accuracy. Specifically, the time series filter 141 determines that the accuracy is high when the difference occurs continuously N times in time series, outputs the case where the difference has occurred as the time series filter output 36, and the trigger determination unit 144 outputs the transmission trigger 35 for saving the corresponding captured image 31. In addition, hereinafter, the time series filter output 36 is also referred to as the "time series score".

[0036] Figure 6This is a diagram showing an example of the detection difference 34 input to the time series filter 141. In Figure 6 , the data for five time instants from time T-2 to time T+2 is shown from top to bottom in the diagram, and the current detection result 32, the new detection result 33, and the detection difference 34 are shown from left to right in the diagram. The slashes in the detection difference 34 indicate that no difference is detected. In this case, when the accuracy threshold is set to N = 3, since there is no difference at time instants T-2, T-1, T+1, and T+2 before and after time T when the difference is generated, it is determined that the accuracy of difference generation is low, and the time series filter 141 ignores the difference at time T. In addition, in this example, assuming that the accuracy threshold is N = 1, the time series filter 141 outputs the case where a difference is generated as the time series filter output 36.

[0037] Figure 7 This is a diagram showing another example of the detection difference 34 input to the time series filter 141. In Figure 7 , the data for six time instants from time T-2 to time T+3 is shown from top to bottom in the diagram, and the current detection result 32, the new detection result 33, and the detection difference 34 are shown from left to right in the diagram. In Figure 7 the example shown, the coordinates of the detection box in the new detection result 33 fluctuate and are different at each time instant from the current detection result 32. In Figure 7 the example shown, the difference extraction unit 13 determines that the difference is above the threshold at all time instants and outputs the detection boxes of the current detection result 32 and the new detection result 33 at each time instant as the detection difference 34.

[0038] The time series filter 141 averages the coordinate fluctuations of the detection boxes in time series, calculates the average of the IoU with the current detection result 32 (hereinafter referred to as "Aiou") and makes a determination. For example, the time series filter 141 can be configured to ignore the difference between the current detection result 32 and the new detection result 33 when Aiou is less than 0.5. The averaging process in this process can be a simple average within a specified time or a moving average. In summary, the time series filter output 36 can be said to be a value that analyzes the accuracy of the detection difference 34 from the perspective of time series. However, in Figure 7 the example shown, the detection difference 34 contains the data required for the calculation of IoU. At this time, if the time series filter output 36 is represented by the symbol Ct, it can be defined as follows, for example.

[0039] Ct = D × Aiou... (Equation 2)

[0040] When the trigger determination unit 144 determines that Ct exceeds a certain determination threshold, it outputs a transmission trigger 35 for saving the captured image 31 corresponding to the calculated detection difference 34. These processes are an example, and it is only necessary to be configured in such a way that the transmission trigger 35 is output according to whether the detection difference 34 occurs stably when observed in time series.

[0041] Figure 8 It is a diagram showing a second example of the transmission determination unit 14. Figure 8 The shown transmission determination unit 14 includes a spatial filter 142 and a trigger determination unit 144. The spatial filter 142 takes the detection difference 34 as an input and outputs a spatial filter output 37. The spatial filter 142 calculates the importance of the generation position of the detection difference 34 and outputs the spatial filter output 37. In addition, hereinafter, the spatial filter output 37 will also be referred to as "position score".

[0042] For example, if it is assumed that the output of the image recognition device 1 is used for a peripheral recognition application of autonomous driving, when a detection difference 34 occurs on the forward path of the vehicle 9, it may be related to control such as sudden braking and sudden steering in the subsequent control program. Therefore, the spatial filter 142 takes the detection difference 34 on the traveling path of the vehicle 9 as an important difference and outputs the spatial filter output 37, and outputs the transmission trigger 35 from the trigger determination unit 144. On the other hand, since the detection difference 34 in the sky, distant buildings, etc. in the captured image 31 has no influence on the control of the vehicle 9, the spatial filter 142 ignores this difference. For the vehicle 9, the important area dynamically changes according to the movement of surrounding objects, positional relationships, road shapes, the speed of the vehicle 9, steering information, etc. Therefore, it is preferable to use these information to define an area corresponding to the influence on control.

[0043] Figure 9 It is a diagram showing an example of the risk determination map 142M referred to by the spatial filter 142. The risk determination map 142M sets the risks A to D for each area. The highest risk A corresponds to the area within the captured image 31 that captures the lane in which the vehicle 9 is traveling and is less than a specified distance. The second highest risk B corresponds to the area within the captured image 31 of the lane adjacent to the lane in which the vehicle 9 is traveling. The third highest risk C corresponds to the area within the captured image 31 that is more than two lanes away from the lane in which the vehicle 9 is traveling, the sidewalk, and is more than a specified distance away.

[0044] The spatial filter 142 outputs, for example, as the spatial filter output 37, the value corresponding to the risk level at the position corresponding to the detection difference 34 in the risk determination map 142M. Specifically, "5" is output when the detection difference 34 exists in the area of risk level A, "3" is output when the detection difference 34 exists in the area of risk level B, "1" is output when the detection difference 34 exists in the area of risk level C, and "0" is output when the detection difference 34 exists in the area of risk level D. However, when the detection difference 34 exists in the area of risk level D, the spatial filter 142 may not output the spatial filter output 37.

[0045] Figure 9 The overlapping image 31M shown at the lower part of overlaps the boundary shown in the risk determination map 142M on the captured image 31. It can be seen that the area of risk level A is the area of the lane in which the vehicle 9 travels, and the area of risk level D is the open area. In addition, in the risk determination map 142M, the boundary B1, which is the boundary between the area of risk level A and the area of risk level C, may also move upward in the figure as the traveling speed of the vehicle 9 increases. This is because the faster the speed of the vehicle 9, the wider the dangerous area. Also, the boundary B2, which is the boundary between the area of risk level A and the area of risk level B, may move and deform according to the steering operation of the vehicle 9 and the shape of the road.

[0046] In order to deform the risk determination map 142M according to the traveling speed of the vehicle 9, the spatial filter 142 can utilize the output of a speed sensor (not shown) and the rotational speed information of the tires. In order to deform the risk determination map 142M according to the steering operation of the vehicle 9, the spatial filter 142 can utilize the steering operation amount of the vehicle 9 and the output of a gyro sensor (not shown). In order to deform the risk determination map 142M according to the shape of the road, the spatial filter 142 can utilize the output of a sensor (not shown) that can measure the shape of the road, a combination of map information and its own position information, etc.

[0047] Figure 10 is a diagram showing a third example of the transmission determination unit 14. Figure 10 The transmission determination unit 14 shown in Figure 10 includes a time series filter 141, a spatial filter 142, a fractional filter 143, and a trigger determination unit 144. The trigger determination unit 144 uses the outputs of the time series filter 141 and the spatial filter 142 quantified by the fractional filter 143 to determine whether the transmission trigger 35 needs to be output.

[0048] The fractional filter 143 includes a differential fraction calculation unit 201 and a fraction storage unit 202. In Figure 10The lower part shows a diagram of an example of data stored in the fractional storage unit 202. The time-series filter output 36 and the spatial filter output 37 are input to the differential fraction calculation unit 201. The differential fraction calculation unit 201 uses the time-series filter output 36 and the spatial filter output 37 to calculate the quantized differential fraction 81 and stores it in the fractional storage unit 202. When the time-series filter output 36 is represented by the differential accuracy Ct(t) and the spatial filter output 37 is represented by the importance P(t), the differential fraction S(t) at time t can be defined as follows, for example.

[0049] S(t) = Ct(t) × P(t) … (Equation 3)

[0050] The trigger determination unit 144 can also generate a transmission trigger 35 for the captured image 31 corresponding to the maximum fraction during a specified period. In addition, the trigger determination unit 144 can also generate a transmission trigger 35 for the captured image 31 corresponding to the differential fraction 81 that is one of multiple maximum values. Furthermore, the trigger determination unit 144 can generate a transmission trigger 35 for the captured image 31 corresponding to the differential fraction 81 greater than the threshold value. Specific description will be made with reference to Figure 10 the diagram shown in the lower part of

[0051] At Figure 10 the lower part shows the time-series differential fraction 81 from time t0 to time t6. From time t0 to time t6, the differential fraction 81 repeatedly increases and decreases, and the maximum values occur at three times, t1, t3, and t4, and the maximum value is at time t4. In addition, the differential fraction 81 exceeds the threshold value ts from time t2 to time t5. The trigger determination unit 144 can also generate a transmission trigger 35 for the captured image 31 at time t4 when the differential fraction 81 is the largest. In addition, the trigger determination unit 144 can also generate a transmission trigger 35 for the captured images 31 at times t1, t3, and t4 when the differential fraction 81 becomes a maximum value. Furthermore, the trigger determination unit 144 can generate a transmission trigger 35 for all the captured images 31 from time t2 to time t5 that have a fraction greater than the threshold value ts.

[0052] According to the first embodiment described above, the following operational effects can be obtained. (1) The image recognition device 1 performs object detection on the captured image 31 using an image recognition program. The image recognition device includes: a ROM 42 that stores the current program 11 as an existing image recognition program and the new program 12 as a new image recognition program; a difference extraction unit 13 that extracts a detection difference 34, which is the difference between the object detection result of the current program 11 and the image recognition result of the new program 12 for the same captured image 31; a transmission determination unit 14 that determines whether it is necessary to save the captured image 31 based on the occurrence status of the detection difference 34; and a trigger reception unit 19 that outputs the captured image 31 determined by the transmission determination unit 14 to be saved to the outside of the image recognition device 1. Therefore, it is possible to appropriately determine the captured image 31 for evaluating the new program 12. Specifically as follows.

[0053] In the current program 11 and the new program 12, the captured images 31 are sequentially input from the camera 91 and image processing is performed, continuously outputting the current detection result 32 and the new detection result 33. Taking them as inputs, the difference extraction unit 13 also continuously outputs the detection difference 34. In the case where an image processing program is configured based on machine learning such as DNN, the detection difference 34 occurs frequently. Therefore, in the case of simply saving the captured image 31, a large storage area is required, and the captured image 31 that is not suitable for evaluating the new program 12 is also included. In the present embodiment, this problem is solved by the transmission determination unit 14 determining an appropriate captured image 31. In addition, since the captured image 31 output by the image recognition device 1 is transmitted to the outside of the vehicle 9 via the vehicle exterior communication device 92, by the transmission determination unit 14 determining an appropriate captured image 31, it is also possible to obtain the effect of reducing the communication volume from the vehicle 9 to the outside.

[0054] (2) The transmission determination unit 14 has a time series filter 141, and the time series filter 141 determines whether it is necessary to save based on the detection differences of a plurality of captured images 31 with different acquisition times. Therefore, as Figure 6 shown, the image recognition device 1 can select and save the continuously generated detection differences 34.

[0055] (3) The transmission determination unit 14 has a spatial filter 142 that determines whether it is necessary to save based on the importance corresponding to the position of the detection difference in the captured image 31. Therefore, as Figure 9 shown, the image recognition device 1 can save the detection differences 34 at important positions in the captured image 31.

[0056] (4) The transmission determination unit 14 includes: a time-series filter 141 that calculates the occurrence of detection differences in a plurality of captured images 31 with different acquisition times as a time-series score; a spatial filter 142 that calculates the importance corresponding to the position of the detection difference in the captured image 31 as a position score; a score filter 143 that calculates a difference score 81 using the time-series score and the position score; and a trigger determination unit 144 that determines whether saving is required based on the difference score 81. Therefore, the image recognition device 1 can select an appropriate captured image 31 from the viewpoints of both time and space.

[0057] (5) The score filter 143 includes a score storage unit 202 that stores the difference score 81, and the trigger determination unit 144 determines whether saving is required based on the difference score 81 stored in the score storage unit 202. Therefore, the image recognition device 1 can select the captured image 31 corresponding to the maximum value or maximum extreme value of the difference score 81.

[0058] (6) The trigger determination unit 144 determines whether saving is required to save at least one of the captured image 31 corresponding to the maximum difference score 81 within a specified period, the captured image 31 corresponding to the difference score 81 exceeding a specified threshold, and the captured image 31 corresponding to the difference score 81 that becomes a maximum extreme value.

[0059] (Modification Example 1) Figure 11 This is a configuration diagram of the image recognition device 1 of Modification Example 1. The image recognition device 1 may also include an in-device storage unit 18 as a non-volatile storage device, and the trigger reception unit 19 stores the captured image to be saved in the in-device storage unit 18. In this case, the vehicle 9 may not include an out-of-vehicle communication device 92. Additionally, in this modification example, since the transmission determination unit 14 determines the captured image 31 stored in the image recognition device 1, the transmission determination unit 14 may be referred to as a "saving determination unit".

[0060] - Second Embodiment - Refer to Figures 12 to 13 A second embodiment of the image recognition device and the saved image determination method will be described. In the following description, the same reference numerals are given to the same components as those in the first embodiment, and the differences will be mainly described. For the content not specifically described, it is the same as that in the first embodiment. In this embodiment, it is mainly different from the first embodiment in that only the captured images that cannot be more appropriately processed by the new program than the current program are transmitted.

[0061] Figure 12It is a configuration diagram of the image recognition device 1A of the second embodiment. In addition to the configuration of the image recognition device 1 of the first embodiment, the image recognition device 1A further includes a verification unit 15. The detection difference 34 is input from the difference extraction unit 13 to the verification unit 15, the reliability score 38 is calculated, and the reliability score 38 is output to the transmission determination unit 14. In addition to the processes described in the first embodiment, the transmission determination unit 14 also outputs a transmission trigger 35 using the reliability score 38.

[0062] The verification unit 15 verifies the detection performance degradation of the new program 12 relative to the current program 11 through a rule base and object detection, and calculates the reliability score 38. The reliability score 38 is, for example, a numerical value from 1 to 10, and the larger the value, the greater the possibility that the detection difference 34 is caused by the degradation of the new program 12 relative to the current program 11. For example, assume the following situation: The new program 12 is modified to newly detect an object R relative to the current program 11, and a detection difference 34 is generated because only the new program 12 recognizes a certain area.

[0063] In this case, when the verification unit 15 performs object detection on the detection difference 34 and detects the object R, since the detection difference 34 is the result of the new program 12 operating as expected, a low reliability score 38 is set. However, when no object is detected as a result of performing object detection on the detection difference 34, since the new program 12 has generated a false detection, a high reliability score 38 is set.

[0064] Figure 13 It is a configuration diagram of the transmission determination unit 14A in the second embodiment. The transmission determination unit 14A includes a time series filter 141, a spatial filter 142, a score filter 143A, and a trigger determination unit 144. The time series filter 141 and the spatial filter 142 are input with the detection difference 34 in the same manner as in the first embodiment. The processes of the time series filter 141 and the spatial filter 142 are the same as those in the first embodiment, so their descriptions are omitted. The score filter 143A includes a performance degradation score calculation unit 301 and a score storage unit 202. The time series filter output 36, the spatial filter output 37, the reliability score 38, and the new detection result 33 are input to the score filter 143A.

[0065] The performance degradation score calculation unit 301 calculates the performance degradation score 82 as shown in Equation 4 below, and outputs the performance degradation score 82 to the score storage unit 202. If the performance degradation score 82 at time t is represented by Sd(t), then Sd(t) can be calculated as follows.

[0066] Sd(t) = Ct(t)P(t)Vc(t) / Dc(t)…(Equation 4)

[0067] In Equation 4, Ct(t) is the output 36 of the time series filter at time t, P(t) is the output 37 of the spatial filter at time t, Vc(t) is the verified reliability score 38 at time t, and Dc(t) is the reliability of the detection result in the new program 12 at time t. That is, the lower the reliability at the time of performance degradation determination, the lower the performance degradation score Sd(t), and the lower the reliability Dc(t) of the detection result in the new program 12, the higher the performance degradation score Sd(t).

[0068] The trigger determination unit 144 processes the time series performance degradation score 82 stored in the score storage unit 202 in the same manner as the differential score 81 in the first embodiment and generates a transmission trigger 35. That is, the trigger determination unit 144 uses the captured image 31 corresponding to the performance degradation score 82 that becomes the maximum value during a specified period, the performance degradation score 82 that becomes an extreme value, the performance degradation score 82 greater than a specified threshold, etc. as the object and generates a transmission trigger 35.

[0069] According to the second embodiment described above, the following effects can be obtained. (7) The image recognition device 1A includes a verification unit 15 that outputs a reliability score 38 indicating the degree to which the detection difference is caused by the degradation of the new program 12 relative to the current program 11. The transmission determination unit 14 determines whether to save based on the occurrence status of the detection difference 34 and the reliability score 38. Therefore, the image recognition device 1A can select the detection difference 34 caused by the new program 12 being worse than the current program 11.

[0070] (8) The new program 12 outputs a new program reliability as the reliability of the detection result. The image recognition device 1A includes: a time series filter 141 that calculates the occurrence of the detection difference for a plurality of captured images 31 at different acquisition times as a time series filter output 36 called a time series score; a spatial filter 142 that calculates the importance corresponding to the position of the detection difference in the captured image 31 as a spatial filter output 37 called a position score; and a performance degradation score calculation unit 301 that calculates a performance degradation score 82 based on the new program reliability, the time series filter output 36, the spatial filter output 37, and the reliability score 38. The transmission determination unit 14A determines whether to save based on the performance degradation score 82.

[0071] (9) The image recognition device 1A includes a score storage unit 202 that stores the performance degradation score 82. The trigger determination unit 144 of the transmission determination unit 14A determines whether to save based on the performance degradation score 82 stored in the score storage unit 202.

[0072] (Modification Example of the Second Embodiment) The performance degradation score calculation unit 301 may also calculate the difference score S(t) as follows without using the reliability score 38 calculated by the verification unit 15.

[0073] S(t)=Ct(t)P(t) / Dc(t)…(Equation 5)

[0074] The subsequent processing and determination of this difference score S(t) are the same as those in the second embodiment, so the description is omitted.

[0075] - Third Embodiment - Refer to Figures 14 to 15 A third embodiment of the image recognition device and the saved image determination method will be described. In the following description, the same reference numerals are given to the same components as in the first embodiment, and the differences will be mainly described. For the content not specifically described, it is the same as in the first embodiment. In this embodiment, it is mainly different from the first embodiment in that the difference between the current program and the new program is judged by comparing the outputs of the subsequent control programs.

[0076] Figure 14 FIG. is a configuration diagram of a vehicle 9B equipped with an image recognition device 1B according to the third embodiment. The vehicle 9B includes an image recognition device 1B, a camera 91, an external vehicle communication device 92, a control program 93, a second control program 93A, and a control difference extraction unit 96. The second control program 93A is the same as the control program 93. However, the difference is that the output of the control program 93 is used for the control of the vehicle 9B, while the output of the second control program 93A is not used for the control of the vehicle 9B. The control difference extraction unit 96 extracts the difference between the output of the control program 93 and the output of the second control program 93A.

[0077] The control program 93 and the second control program 93A may be different instances generated from the same binary file, or the same instance may be processed as different instances in a time-sharing manner as a disguise. However, the control program 93 and the second control program 93A do not need to be exactly the same at the binary level, as long as it is confirmed that the difference in the output is caused by the difference in the input.

[0078] Figure 15This is a diagram showing the relationship between the image recognition device 1B, the control program 93, the second control program 93A, and the control difference extraction unit 96. The current program 11 outputs the current detection result 32 to the difference extraction unit 13 and the control program 93, which is the same as in the first embodiment. The new program 12 outputs the new detection result 33 not only to the difference extraction unit 13 but also to the second control program 93A. The current detection result 32 is input to the control program 93, and the first operation result 94 is output. The new detection result 33 is input to the second control program 93A, and the second operation result 94A is output. The control difference extraction unit 96 extracts the difference between the first operation result 94 and the second operation result 94A, and outputs the control difference 95 to the transmission determination unit 14.

[0079] The detection difference 34 and the control difference 95 are input to the transmission determination unit 14. The transmission determination unit 14 can also output the transmission trigger 35 under these AND conditions. In addition, the transmission determination unit 14 can not only use the time-series filter output 36 and the spatial filter output 37, but also use the control difference 95 to calculate the difference score 81, and generate the transmission trigger 35 for the captured image 31 corresponding to the maximum value, extreme value, and value greater than the threshold of the difference score 81.

[0080] According to the above third embodiment, the following effects can be obtained. (10) The current detection result 32, which is the object detection result of the current program 11, is used in the control program 93 that outputs the first operation result 94. The first operation result 94 is used for the control of the vehicle 9 equipped with the image recognition device 1. The transmission determination unit 14 determines whether to save according to the control difference 95, which is the difference between the second operation result 94A obtained by inputting the new detection result 33, which is the object detection result of the new program 12, into the second control program 93A, and the first operation result 94, and the occurrence status of the detection difference 34. Therefore, by comparing the differences between the current program 11 and the new program 12 from the perspective of the output of the control program 93 that uses the output of the current program 11, the captured image 31 that truly affects the overall system can be screened out.

[0081] The first embodiment to the third embodiment and their respective modification examples described above can be executed separately or in combination. In addition, according to the system request and the verification stage, the sensitivity of collecting differential data may vary, or the upper limit of the frame rate of the images that can be transmitted may change due to communication cost constraints. Therefore, the configuration of each embodiment can also be switched according to conditions for operation. For example, it is also possible to obtain the configuration of the third embodiment immediately after updating the program and starting the prototype verification, extract the differential information that is truly important for control, and then collect the differential data of the conditions to be concerned while adjusting various parameters in the configurations of the first embodiment and the second embodiment.

[0082] In addition, not only can the highly valuable captured images 31 obtained in each embodiment be sent or saved, but they can also be used to improve the accuracy of the current detection result 32 for vehicle control. For example, when there is a detection error in the current detection result 32 and the new detection result 33 of the updated version is correct (when the performance of the updated version is improved), and when it is determined in the transmission determination unit that the accuracy is high, by inputting the verified result into the subsequent control program 93, a driving system with higher performance can also be achieved.

[0083] In the above-described embodiments and modification examples, the configuration of the functional blocks is merely an example. Several functional structures represented as different functional blocks can also be integrally formed, or the configuration represented by one functional block diagram can be divided into two or more functions. In addition, a part of the functions possessed by each functional block can also be the configuration possessed by other functional blocks.

[0084] In the above-described embodiments and modification examples, the programs for implementing the difference extraction unit 13 and the transmission determination unit 14 are stored in the ROM 42, but the programs can also be stored in a non-volatile storage device. In addition, the image recognition device 1 can also be provided with an input / output interface (not shown), and when necessary, the input / output interface and the image recognition device 1 read the program from other devices via an available medium. Here, the medium refers to, for example, a storage medium that can be attached and detached to the input / output interface, or a communication medium, that is, a wired, wireless, optical, etc. network, or a carrier wave or digital signal propagated in the network. In addition, part or all of the functions implemented by the program can also be implemented by a hardware circuit or an FPGA.

[0085] The above-described embodiments and modification examples can also be combined respectively. In the above, various embodiments and modification examples have been described, but the present invention is not limited to these contents. Other modes considered within the technical idea of the present invention are also included within the scope of the present invention. Reference Signs

[0086] 1, 1A, 1B: Image recognition devices, 9, 9B: Vehicles, 11: Current program, 12: New program, 13: Difference extraction unit, 14, 14A: Transmission determination unit, 15: Verification unit, 18: In-device storage unit, 31: Captured image, 32: Current detection result, 33: New detection result, 34: Detection difference, 35: Transmission trigger, 36: Time-series filter output, 37: Spatial filter output, 38: Reliability score, 81: Difference score, 82: Performance degradation score, 91: Camera, 93: Control program, 93A: Second control program, 94: First operation result, 94A: Second operation result, 95: Control difference, 96: Control difference extraction unit, 141: Time-series filter, 142: Spatial filter, 142M: Hazard determination map, 143: Score filter, 143A: Score filter, 144: Trigger determination unit, 201: Difference score calculation unit, 202: Score storage unit, 301: Performance degradation score calculation unit.

Claims

1. An image recognition device that performs object detection on an input image using an image recognition program, wherein the image recognition device is characterized by comprising: a storage unit that stores a current program as the existing image recognition program and a new program as the new image recognition program; a difference extraction unit that extracts a difference, i.e., a detection difference, between the object detection result of the current program and the image recognition result of the new program for the same input image; a transmission determination unit that determines whether to save the input image based on the occurrence status of the detection difference; and a saving unit that outputs the input image determined by the transmission determination unit to be saved to the outside of the image recognition device, or saves the input image determined by the transmission determination unit to be saved in the image recognition device.

2. The image recognition device according to claim 1, wherein the transmission determination unit has a time series filter that determines the need for saving based on the detection differences for a plurality of input images having different acquisition times.

3. The image recognition device according to claim 1, wherein the transmission determination unit has a spatial filter that determines the need for saving based on the importance corresponding to the position of the detection difference in the input image.

4. The image recognition device according to claim 1, wherein the transmission determination unit comprises: a time series filter that calculates the occurrence of the detection differences for a plurality of input images having different acquisition times as a time series score; a spatial filter that calculates the importance corresponding to the position of the detection difference in the input image as a position score; a score filter that calculates a difference score using the time series score and the position score; and a trigger determination unit that determines the need for saving based on the difference score.

5. The image recognition device according to claim 4, wherein the score filter further comprises a score storage unit that stores the difference score, and the trigger determination unit determines the need for saving based on the difference score stored in the score storage unit.

6. The image recognition device according to claim 5, wherein the trigger determination unit determines the need for saving to save at least one of the input image corresponding to the maximum difference score within a specified period, the input image corresponding to the difference score exceeding a specified threshold, and the input image corresponding to the difference score that becomes a maximum value.

7. The image recognition device according to claim 1, wherein the image recognition device further comprises a verification unit that outputs a reliability score indicating the degree to which the detection difference is caused by the deterioration of the new program relative to the current program, and the transmission determination unit determines the need for saving based on the occurrence status of the detection difference and the reliability score.

8. The image recognition device according to claim 7, wherein The new program also outputs a new program reliability which is the reliability of the detection result. The image recognition device includes: a time series filter that calculates the occurrence of the detection difference for a plurality of the input images at different acquisition times as a time series score; a spatial filter that calculates the importance corresponding to the position of the detection difference in the input image as a position score; and a performance degradation score calculation unit that calculates a performance degradation score based on the new program reliability, the time series score, the position score, and the reliability score. The transmission determination unit determines whether the saving is necessary based on the performance degradation score.

9. The image recognition device according to claim 8, wherein the image recognition device further includes a score storage unit that stores the performance degradation score, and the transmission determination unit determines whether the saving is necessary based on the performance degradation score stored in the score storage unit.

10. The image recognition device according to claim 1, wherein the object detection result of the current program is used in a first program that outputs a first operation result, the first operation result is used for the control of a vehicle on which the image recognition device is mounted, and the transmission determination unit determines whether the saving is necessary based on the difference between the second operation result obtained by inputting the object detection result of the new program into the first program, i.e., the control difference, and the first operation result, and the occurrence status of the detection difference.

11. The image recognition device according to claim 7, wherein the object detection result of the current program is used in a first program that outputs a first operation result, the first operation result is used for the control of a vehicle on which the image recognition device is mounted, and the transmission determination unit determines whether the saving is necessary based on the difference between the second operation result obtained by inputting the object detection result of the new program into the first program, i.e., the control difference, and the first operation result, and the occurrence status of the detection difference.

12. A method for determining a saved image, which is executed by an image recognition device that performs object detection on an input image using an image recognition program. The method for determining a saved image is characterized in that the image recognition device includes a storage unit that stores a current program as the existing image recognition program and a new program as the new image recognition program, The described method for determining the saved image includes: a difference extraction step of extracting a difference, i.e., a detection difference, between the object detection result of the current program and the image recognition result of the new program for the same input image; a transmission determination step of determining whether it is necessary to save the input image based on the occurrence status of the detection difference; and a saving step of outputting the input image determined to be saved in the transmission determination step to the outside of the image recognition device, or saving the input image determined to be saved in the transmission determination step in the image recognition device.

Images