Image recognition device and method for determining images to be saved

Abstract

To appropriately determine an image for evaluating a newly created program.SOLUTION: An image recognition device performs object detection using an image recognition program with respect to an input image. The image recognition device comprises: a storage section that stores a current program, which is a conventional image recognition program, and a new program, which is a new image recognition program; a difference extraction section that extracts a detection difference which is a difference between an object detection result of the current program and an image recognition result of the new program with respect to the same input image; a transmission determination section that determines whether or not to save the input image on the basis of an appearance situation of the detection difference; and a saving section that outputs the input image that the transmission determination section has determined to save to the outside of the image recognition device, or that saves the input image that the transmission determination section has determined to save in the image recognition device.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an image recognition device and a method for determining an image to be saved. [Background technology]

[0002] In applications where a program defect could cause a serious accident resulting in loss of life, thorough performance verification is required before the program is put into practical use. For example, in the case of automobile programs, verification is performed using a huge number of test patterns in a virtual development environment such as a simulator, followed by hundreds of thousands of kilometers of test runs on public roads in an experimental vehicle to confirm the absence of defects and ensure safety. Verification using an experimental vehicle is also required when updating a program, but verification using an actual device operating in a real environment poses challenges, including the enormous time and human costs required to verify the results. Patent Document 1 describes a method for dividing a first program including multiple control processes and a second program including multiple control processes in which at least some of the control processes have been changed, into a parallel process that realizes parallel processing of the multiple control processes and a functional sequential process that realizes the function of the multiple control processes, and outputting the first program in which each of the multiple control processes has been divided into the parallel process and the functional sequential process as a first divided program, and outputting a program in which each of the multiple control processes has been divided into the parallel process and the functional sequential process. An operation verification device is disclosed that includes a division unit that outputs the second program as a second divided program, and a cause estimation unit that, when a functional defect is detected in the second divided program as a functional defect, infers that different functional sequential processing between the first divided program and the second divided program is the cause of the functional defect, and, when a defect caused by parallel processing in the second divided program is detected as a concurrency defect, infers that different parallel processing between the first divided program and the second divided program is the cause of the concurrency defect. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2018 / 150504 Summary of the Invention [Problem to be solved by the invention]

[0004] The invention described in Patent Document 1 cannot properly determine images for evaluating a newly created program. [Means for solving the problem]

[0005] An image recognition device according to a first aspect of the present invention is an image recognition device that performs object detection on an input image using an image recognition program, and includes a memory unit that stores a current program, which is the conventional image recognition program, and a new program, which is the new image recognition program; a difference extraction unit that extracts a detection difference, which is the difference between the object detection result of the current program and the image recognition result of the new program for the same input image; a transmission determination unit that determines whether or not to save the input image based on the appearance status of the detection difference; and a storage unit that outputs the input image that the transmission determination unit determines to save to the outside of the image recognition device, or stores the input image that the transmission determination unit determines to save in the image recognition device. A method for determining an image to be saved according to a second aspect of the present invention is a method for determining an image to be saved executed by an image recognition device that performs object detection on an input image using an image recognition program, wherein the image recognition device has a memory unit that stores a current program, which is the conventional image recognition program, and a new program, which is the new image recognition program, and includes a difference extraction step that extracts a detection difference, which is the difference between the object detection result of the current program and the image recognition result of the new program for the same input image, a transmission determination step that determines whether or not the input image needs to be saved based on the appearance status of the detection difference, and a storage step that outputs the input image that is determined to be saved by the transmission determination step to an outside of the image recognition device, or stores the input image that is determined to be saved by the transmission determination step in the image recognition device. [Effects of the Invention]

[0006] According to the present invention, images for evaluating a newly created program can be appropriately determined. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a diagram illustrating a configuration of a vehicle equipped with an image recognition device according to a first embodiment. [Figure 2] 1 is a diagram illustrating the configuration of an image recognition device according to a first embodiment. [Figure 3] Hardware configuration diagram of image recognition device [Figure 4] Schematic diagram showing the processing of the differential extraction unit [Figure 5] FIG. 10 is a diagram illustrating a first example of a transmission determination unit. [Figure 6] FIG. 10 is a diagram showing an example of a detected difference input to a time series filter. [Figure 7] FIG. 10 is a diagram showing another example of detected differences input to the time series filter. [Figure 8] FIG. 2 is a diagram illustrating a second example of a transmission determination unit. [Figure 9] FIG. 10 is a diagram showing an example of a risk determination map referenced by a spatial filter; [Figure 10]FIG. 10 is a diagram illustrating a third example of a transmission determination unit. [Figure 11] Configuration diagram of an image recognition device in Modification 1 [Figure 12] 1 is a diagram illustrating a configuration of an image recognition device according to a second embodiment. [Figure 13] 10 is a diagram illustrating a configuration of a transmission determination unit according to a second embodiment. [Figure 14] 10 is a diagram showing the configuration of a vehicle equipped with an image recognition device according to a third embodiment. [Figure 15] FIG. 1 is a diagram showing the relationship between an image recognition device, a control program, a second control program, and a control difference extraction unit. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Each embodiment is an example for explaining the present invention, and for clarity of explanation, appropriate omissions and simplifications have been made. The present invention can be implemented in various other forms. Unless otherwise specified, each component may be singular or plural.

[0009] The position, size, shape, range, etc. of each component shown in the drawings may not represent the actual position, size, shape, range, etc. in order to facilitate understanding of the invention. Therefore, the present invention is not necessarily limited to the position, size, shape, range, etc. disclosed in the drawings. When there are multiple components having the same or similar functions, they may be described using the same reference numeral with different subscripts. Furthermore, when it is not necessary to distinguish between these multiple components, the subscripts may be omitted in the description.

[0010] In each embodiment, there may be a description of processing performed by executing a program. Here, a computer executes the program using a processor (e.g., CPU, GPU), and performs processing defined by the program using storage resources (e.g., memory), interface devices (e.g., communication ports), etc. Therefore, the entity that executes the program and performs the processing may be the processor. Similarly, the entity that executes the program and performs the processing may be a controller, device, system, computer, or node that has a processor. The entity that executes the program and performs the processing may be a calculation unit, and may include a dedicated circuit that performs specific processing. Here, the dedicated circuit may be, for example, an FPGA (Field Programmable Gate Array), ASIC (Application Specific Integrated Circuit), CPLD (Complex Programmable Logic Devices), etc.

[0011] A program may be installed on a computer from a program source. The program source may be, for example, a program distribution server or a computer-readable storage medium. When the program source is a program distribution server, the program distribution server may include a processor and storage resources for storing the program to be distributed, and the processor of the program distribution server may distribute the program to be distributed to other computers. In addition, in embodiments, two or more programs may be realized as one program, or one program may be realized as two or more programs.

[0012] In recent years, in order to realize highly automated driving, peripheral recognition applications using image recognition programs that apply deep neural networks (DNN), a type of machine learning, have become increasingly popular.Compared to rule-based algorithms, programs that use machine learning are known to be less susceptible to performance changes when the program is updated.

[0013] For example, in a peripheral recognition application, the current version of the program (hereinafter referred to as the "current program") may correctly recognize an object in a certain image, but the updated version of the program (hereinafter referred to as the "new program") may have deteriorated object recognition performance. Deterioration in recognition performance may include the recognition of an object with a misaligned position, or the failure to detect the object because the reliability score used to determine recognition is below a threshold.

[0014] These phenomena occur due to the complex interaction of various parameters, including the size, color, orientation, shape, and other characteristics of the object in the image, as well as its relative position to surrounding objects and the background, and even hardware-induced noise that is mixed in when the image is taken.As a result, the detection results fluctuate each time image recognition processing is performed, and so when the processing results of the current program and the new program are subjected to equivalence verification, differences between the two programs very frequently occur.

[0015] For the purpose of post-mortem verification and program development, it is useful to save images that show different processing results between the current program and the new program. However, if images are saved solely because of differences in processing results, the number of images to be saved will be enormous. In other words, if images with differences in recognition performance between the current program and the new program are saved, the number of images to be saved will be extremely large. Below, we will explain a method for determining which images to save.

[0016] In the following embodiments, an example will be described in which the present invention is applied to an in-vehicle ECU for vehicle control, for example, an Advanced Driver Assistance System (ADAS) or Autonomous Driving (AD). However, the present invention is not limited to in-vehicle ECUs for ADAS and AD. The present invention may also be used to verify updates to peripheral recognition AI for autonomous guided vehicles (AGVs) and construction machinery, or to verify updates to AI for surveillance cameras, etc. The present invention is generally applicable to verifying updates to information processing algorithms that use machine learning, such as image processing.

[0017] -First embodiment- Hereinafter, a first embodiment of an image recognition device and a method for determining an image to be saved will be described with reference to FIGS.

[0018] FIG. 1 is a configuration diagram of a vehicle 9 equipped with an image recognition device 1. The vehicle 9 includes the image recognition device 1, a camera 91, an exterior communication device 92, and a control program 93. The image recognition device 1 can communicate with the camera 91, the exterior communication device 92, and the control program 93 using a known communication method. This communication may be wired or wireless. Examples of this communication include IEEE802.3, Controller Area Network, and IEEE802.11. The camera 91 captures images of the surroundings of the vehicle 9 and outputs the captured images (hereinafter referred to as "captured images" or "input images") to the image recognition device 1. The image recognition device 1 performs object detection on the captured images using an image recognition program and outputs the detection results to the control program 93. The image recognition device 1 also outputs some of the captured images to the exterior communication device 92, as described below.

[0019] The exterior communication device 92 transmits the captured images output by the image recognition device 1 via wireless communication to an image storage server 99 located outside the vehicle 9. The exterior communication device 92 and the image storage server 99 may communicate directly wirelessly, or there may be a fixed communication relay point on the ground such as a base station between them, or communication may be via another vehicle. The image storage server 99 stores the captured images received from the exterior communication device 92 in a non-volatile storage device (not shown).

[0020] The control program 93 performs calculations using the detection results output by the image recognition device 1. The content of the calculations executed by the control program 93 is arbitrary, and for example, the control program 93 may control the vehicle 9 based on the detection results, or may notify the driver of the vehicle 9 of the presence of an obstacle based on the calculation results.

[0021] FIG. 2 is a configuration diagram of the image recognition device 1. The image recognition device 1 includes a current program 11, a new program 12, a difference extraction unit 13, a transmission determination unit 14, and a trigger reception unit 19. The camera 91 outputs a captured image 31 to the current program 11, the new program 12, and the trigger reception unit 19. The current program 11 receives at least the captured image 31 as input and outputs a current detection result 32. The new program 12 receives at least the captured image 31 as input and outputs a new detection result 33. The difference extraction unit 13 receives the current detection result 32 and the new detection result as input and outputs a detected difference 34. The transmission determination unit 14 receives the detected difference 34 as input and outputs a transmission trigger 35. The trigger reception unit 19 receives the captured image 31 and the transmission trigger 35 as input and outputs the captured image 31. The trigger reception unit 19 outputs the captured image 31 to the outside for storage, and can therefore also be called a "storage unit."

[0022] Both the current program 11 and the new program 12 are image recognition programs that input a captured image 31, and both detect objects. However, the operational stability of the current program 11 has already been confirmed, whereas the operational confirmation of the new program 12 has not been sufficient. The new program 12 is, for example, a program that corrects oversights or incorrect detections in the current program 11, or a program that has been modified so that it can recognize additional types of objects. The details of the operation of the current program 11 and the new program 12 are not particularly limited, and they may be DNN-based programs that use machine learning, or logic-based programs that apply pattern recognition technology, etc.

[0023] Each of the current program 11 and the new program 12 can take various forms, such as a single binary file, a combination of a binary file and a setting file, a combination of a binary file and a library, etc. Furthermore, at least one of the current program 11 and the new program 12 may be a rewritable logic circuit.

[0024] The current detection result 32 output by the current program 11 and the new detection result 33 output by the new program 12 are the coordinates of an object in the captured image 31, the type of object, and the detection reliability. The object coordinates are, for example, coordinates in an orthogonal coordinate system with the upper left corner of the captured image as the origin. The type of object is, for example, a car, a pedestrian, a bicycle, etc. The current detection result 32 is output to the control program 93, which is its original purpose, and is also output to the difference extraction unit 13 for comparison.

[0025] Since the new detection result 33 output by the new program 12 is an output result of the new program 12 that has not been sufficiently verified, it is not used as input for subsequent image recognition processing in the vehicle ECU, but is used solely as information for verification.

[0026] The difference extraction unit 13 extracts the difference between the current detection result 32 and the new detection result 33 and outputs it as a detected difference 34. Detailed processing by the difference extraction unit 13 will be described later. The transmission determination unit 14 determines whether or not to save the captured image 31 based on the appearance status of the detected difference 34, and outputs a transmission trigger 35. Here, the appearance status of the detected difference 34 refers to, for example, the appearance frequency, appearance continuity, and positional fluctuation of the detected difference 34 in a chronological order. It may also be determined whether the position where the detected difference 34 occurs is within a predetermined area, or it may combine the aforementioned judgment conditions such as appearance frequency and appearance continuity with the judgment condition for the difference occurrence position. The transmission trigger 35 may be a signal that is output only when the transmission determination unit 14 determines that the image should be saved, or it may be a signal that is always output and includes data indicating whether or not saving is required. Detailed processing by the transmission determination unit 14 will be described later.

[0027] The trigger receiving unit 19 outputs the captured image 31 to the exterior communication device 92 based on the transmission trigger 35. In the case where the transmission determination unit 14 outputs the transmission trigger 35 only when it determines that the captured image 31 should be saved, the trigger receiving unit 19 outputs the captured image 31 when the transmission trigger 35 is input. In the case where the transmission determination unit 14 outputs the transmission trigger 35 including data indicating whether saving is required, the trigger receiving unit 19 outputs the captured image 31 when the transmission trigger 35 includes data indicating that saving is required.

[0028] 3 is a hardware configuration diagram of the image recognition device 1. The image recognition device 1 is an electronic control unit (ECU) that includes a CPU 41, which is a central processing unit, a ROM 42, which is a read-only storage device, a RAM 43, which is a readable and writable storage device, and an in-vehicle communication device 44. The CPU 41 loads a program stored in the ROM 42 into the RAM 43 and executes it to perform the various calculations described above.

[0029] The image recognition device 1 may be realized by an FPGA, which is a rewritable logic circuit, or an ASIC (Application Specific Integrated Circuit), which is an application-specific integrated circuit, instead of the combination of the GPU 40, CPU 41, ROM 42, and RAM 43. Furthermore, the image recognition device 1 may be realized by a different combination of configurations, for example, a combination of the CPU 41, ROM 42, RAM 43, and FPGA, instead of the combination of the GPU 40, CPU 41, ROM 42, and RAM 43. Furthermore, a dedicated circuit (such as an AI accelerator) for efficiently executing DNN may be installed instead of the GPU 40. The in-vehicle communication device 44 is compatible with IEEE802.3 or a Controller Area Network and realizes communication with the camera 91 and the out-vehicle communication device 92.

[0030] 3 shows only one GPU 40, one CPU 41, and one RAM 43, but two of each may be installed and the current program 11 and the new program 12 may be executed by different hardware resources. Furthermore, the image recognition device 1 may be configured with multiple ECUs, and the current program 11 and the new program 12 may be executed by different ECUs.

[0031] FIG. 4 is a schematic diagram showing the processing of the difference extraction unit 13. In FIG. 4, the processing results of the captured image 31 shown at the top will be explained. In this captured image 31, another vehicle traveling ahead of the vehicle 9 is captured in the upper left, and the road surface is captured in the rest of the image except for the upper left. The current program 11 and new program 12 process this captured image 31 and output the current detection result 32 and the new detection result 33, respectively. In the current detection result 32, the vehicle area is detected as shown by the dashed line, and in the new detection result 33, the road surface is detected. Hereinafter, the outer edges of the areas detected by the current detection result 32 and the new detection result 33 will be referred to as the "detection frame."

[0032] The difference extraction unit 13 extracts as differences discrepancies in the coordinates where objects are detected between the current detection result 32 and the new detection result 33, discrepancies in the object type, and discrepancies in the detection reliability. Whether the coordinates match can be determined, for example, by whether the IoU (Intersect of Union) targeting the detection frame is equal to or less than a predetermined threshold, for example, 0.5. Specifically, if the area of ​​the detection frame in the current detection result 32 is A32, the area of ​​the detection frame in the new detection result 33 is A33, and the area where both overlap is B, the difference extraction unit 13 determines that the coordinates match if the following formula 1 is satisfied:

[0033] IoU = B / (A32+A33-B)>0.5 (Equation 1)

[0034] 4, the detection frame of the current detection result 32 and the detection frame of the new detection result 33 do not overlap at all, so B in formula 1 is zero, and the condition of formula 1 does not hold, so the difference extraction unit 13 determines that the coordinates do not match. Then, the difference extraction unit 13 outputs the areas of the respective detection frames that do not match as detection differences 34, as shown in the lower part of FIG.

[0035] FIG. 5 is a diagram illustrating a first example of the transmission determination unit 14. The transmission determination unit 14 illustrated in FIG. 5 includes a time series filter 141 and a trigger determination unit 144. The time series filter 141 receives the detected difference 34 as input and outputs a time series filter output 36. The time series filter 141 observes the occurrence of the detected difference 34 in time series and quantifies the degree of accuracy. Specifically, if a difference occurs N times consecutively in time series, the time series filter 141 determines that the accuracy is high, outputs a time series filter output 36 indicating that a difference has occurred, and outputs a transmission trigger 35 for the trigger determination unit 144 to save the corresponding captured image 31. Note that the time series filter output 36 will also be referred to as a "time series score" below.

[0036] FIG. 6 is a diagram showing an example of a detected difference 34 input to the time series filter 141. In FIG. 6, data for five times, from time T-2 to time T+2, is shown from top to bottom, with a current detection result 32, a new detection result 33, and a detected difference 34 shown from left to right. The diagonal lines in the detected difference 34 indicate that no difference was detected. In this case, if the accuracy threshold is set to N=3, there are no differences at times T-2, T-1, T+1, and T+2, which are before and after the time T at which the difference occurred, so the accuracy of the occurrence of a difference is determined to be low, and the time series filter 141 ignores the difference at time T. Note that if the accuracy threshold were N=1 in this example, the time series filter 141 would output a time series filter output 36 indicating that a difference occurred.

[0037] 7 is a diagram showing another example of the detection difference 34 input to the time series filter 141. FIG. 7 shows data for six times from time T-2 to time T+3 from top to bottom, with the current detection result 32, new detection result 33, and detection difference 34 shown from left to right. In the example shown in FIG. 7, the coordinates of the detection frame in the new detection result 33 fluctuate and vary from time to time, and are different from the current detection result 32. In the example shown in FIG. 7, the difference extraction unit 13 determined that the difference was greater than or equal to the threshold at all times, and output the detection frames of the current detection result 32 and the new detection result 33 at each time as the detection difference 34.

[0038] The time series filter 141 averages the coordinate fluctuations of the detection frame over time, and calculates and determines the average IoU (hereinafter referred to as "Aiou") with the current detection result 32. For example, the time series filter 141 may be configured to ignore the difference between the current detection result 32 and the new detection result 33 if Aiou is smaller than 0.5. The averaging process in this process may be a simple average within a predetermined time period, or a moving average. In summary, the time series filter output 36 can be said to be a value obtained by analyzing the accuracy of the detection difference 34 from a time series perspective. However, in the example shown in Figure 7, the detection difference 34 includes data necessary for calculating the IoU. In this case, if the time series filter output 36 is represented by the symbol Ct, it can be defined, for example, as follows:

[0039] Ct=D×Aiou (Formula 2)

[0040] If the trigger determination unit 144 determines that Ct exceeds a certain determination threshold, it outputs a transmission trigger 35 for saving the captured image 31 corresponding to the calculated detection difference 34. These processes are just an example, and it is sufficient that the transmission trigger 35 is output depending on whether the detection difference 34 is occurring stably when observed in time series.

[0041] Fig. 8 is a diagram showing a second example of the transmission determination unit 14. The transmission determination unit 14 shown in Fig. 8 includes a spatial filter 142 and a trigger determination unit 144. The spatial filter 142 receives the detection difference 34 as input and outputs a spatial filter output 37. The spatial filter 142 calculates the importance of the position where the detection difference 34 occurs and outputs the spatial filter output 37. Note that hereinafter, the spatial filter output 37 is also referred to as a "position score."

[0042] For example, assuming that the output of the image recognition device 1 is used in a periphery recognition application for autonomous driving, if a detection difference 34 occurs on the path of the vehicle 9, this may lead to control such as sudden braking or abrupt steering in a subsequent control program. Therefore, the spatial filter 142 outputs the spatial filter output 37 by treating the detection difference 34 on the path of the vehicle 9 as an important difference, and the trigger determination unit 144 outputs a transmission trigger 35. On the other hand, since the detection difference 34 in the sky or a distant building in the captured image 31 has no effect on the control of the vehicle 9, the spatial filter 142 ignores this difference. Since the important area for the vehicle 9 changes dynamically depending on the movement and positional relationship of surrounding objects, the shape of the road, the speed and steering information of the vehicle 9, etc., it is preferable to define the area according to the impact on control using this information.

[0043] 9 is a diagram showing an example of a risk determination map 142M referenced by the spatial filter 142. In the risk determination map 142M, risk levels A to D are set for each region. Risk level A, which has the highest risk level, corresponds to a region in the captured image 31 in which the lane in which the vehicle 9 is traveling is captured at a distance less than a predetermined distance. Risk level B, which has the next highest risk level, corresponds to a region in the captured image 31 in which a lane adjacent to the lane in which the vehicle 9 is traveling is captured. Risk level C, which has the third highest risk level, corresponds to a lane or sidewalk two or more away from the lane in which the vehicle 9 is traveling, or a region in the captured image 31 at least a predetermined distance away.

[0044] For example, the spatial filter 142 outputs a value corresponding to the risk level at the position corresponding to the detection difference 34 in the risk determination map 142M as the spatial filter output 37. Specifically, if the detection difference 34 is in an area of ​​risk level A, the spatial filter 142 outputs "5", if the detection difference 34 is in an area of ​​risk level B, the spatial filter 142 outputs "3", if the detection difference 34 is in an area of ​​risk level C, the spatial filter 142 outputs "1", if the detection difference 34 is in an area of ​​risk level D, the spatial filter 142 does not have to output the spatial filter output 37.

[0045] The superimposed image 31M shown in the lower part of FIG. 9 has the boundary shown in the risk level determination map 142M superimposed on the captured image 31. It can be seen that the area with risk level A is the lane area on which the vehicle 9 is traveling, and the area with risk level D is an empty area. Note that in the risk level determination map 142M, the boundary B1 between the area with risk level A and the area with risk level C may be moved upward in the figure as the traveling speed of the vehicle 9 increases. This is because the faster the speed of the vehicle 9, the larger the dangerous area becomes. Furthermore, the boundary B2 between the area with risk level A and the area with risk level B may be moved or deformed in accordance with the steering operation of the vehicle 9 or the shape of the road.

[0046] The spatial filter 142 can use the output of a speed sensor (not shown) and tire rotation speed information to modify the risk determination map 142M in accordance with the traveling speed of the vehicle 9. The spatial filter 142 can use the steering operation amount of the vehicle 9 and the output of a gyro sensor (not shown) to modify the risk determination map 142M in accordance with the steering operation of the vehicle 9. The spatial filter 142 can use the output of a sensor (not shown) that can measure the shape of the road, a combination of map information and self-position information, etc. to modify the risk determination map 142M in accordance with the shape of the road.

[0047] Fig. 10 is a diagram showing a third example of the transmission determination unit 14. The transmission determination unit 14 shown in Fig. 10 includes a time series filter 141, a spatial filter 142, a score filter 143, and a trigger determination unit 144. The trigger determination unit 144 determines whether or not to output a transmission trigger 35 using the outputs of the time series filter 141 and the spatial filter 142 quantified by the score filter 143.

[0048] The score filter 143 includes a difference score calculation unit 201 and a score storage unit 202. An example of data stored in the score storage unit 202 is shown in the form of a graph at the bottom of Fig. 10. The time series filter output 36 and the spatial filter output 37 are input to the difference score calculation unit 201. The difference score calculation unit 201 calculates a quantified difference score 81 using the time series filter output 36 and the spatial filter output 37 and stores the quantified difference score 81 in the score storage unit 202. The difference score S(t) at time t can be defined as follows, for example, when the time series filter output 36 is represented by a difference probability Ct(t) and the spatial filter output 37 is represented by importance P(t):

[0049] S(t)=Ct(t)×P(t) (Equation 3)

[0050] The trigger determination unit 144 may generate a transmission trigger 35 for a captured image 31 corresponding to the maximum score in a predetermined period. The trigger determination unit 144 may also generate a transmission trigger 35 for a captured image 31 corresponding to a difference score 81 that is a plurality of maximum values. Furthermore, the trigger determination unit 144 may generate a transmission trigger 35 for a captured image 31 corresponding to a difference score 81 that is greater than a threshold value. This will be described in detail with reference to the graph shown at the bottom of FIG.

[0051] The bottom of FIG. 10 shows the difference score 81 in chronological order from time t0 to time t6. The difference score 81 repeatedly increases and decreases from time t0 to time t6, with maximum values ​​at three times, t1, t3, and t4, and the maximum value at time t4. The difference score 81 exceeds the threshold value ts from time t2 to time t5. The trigger determination unit 144 may generate a transmission trigger 35 for the captured image 31 at time t4 when the difference score 81 is maximum. The trigger determination unit 144 may also generate a transmission trigger 35 for the captured image 31 at times t1, t3, and t4 when the difference score 81 reaches its maximum value. Furthermore, the trigger determination unit 144 may generate a transmission trigger 35 for all captured images 31 at times t2 to t5 that have scores greater than the threshold value ts.

[0052] According to the above-described first embodiment, the following advantageous effects can be obtained. (1) The image recognition device 1 performs object detection on a captured image 31 using an image recognition program. The device includes a ROM 42 that stores a current program 11, which is a conventional image recognition program, and a new program 12, which is a new image recognition program; a difference extraction unit 13 that extracts a detection difference 34, which is the difference between the object detection result of the current program 11 and the image recognition result of the new program 12 for the same captured image 31; a transmission determination unit 14 that determines whether or not to save the captured image 31 based on the appearance status of the detection difference 34; and a trigger receiving unit 19 that outputs the captured image 31 that the transmission determination unit 14 determines to be saved to the outside of the image recognition device 1. This allows the device to appropriately determine the captured image 31 for evaluating the new program 12. Specifically, the device is as follows.

[0053] In the current program 11 and the new program 12, captured images 31 are successively input from the camera 91, image processing is performed, and the current detection result 32 and the new detection result 33 are continuously output. The difference extraction unit 13 also continuously outputs detected differences 34 using these as inputs. When the image processing program is configured based on machine learning such as DNN, detected differences 34 occur frequently. Therefore, simply storing the captured images 31 requires a huge amount of storage space, and some captured images 31 that are not appropriate for evaluation of the new program 12 will be included. In this embodiment, this problem is solved by having the transmission determination unit 14 determine appropriate captured images 31. Furthermore, since the captured images 31 output by the image recognition device 1 are transmitted to the outside of the vehicle 9 via the external communication device 92, the transmission determination unit 14's determination of appropriate captured images 31 also has the effect of reducing the amount of communication from the vehicle 9 to the outside.

[0054] (2) The transmission determination unit 14 has a time series filter 141 that determines whether or not to store the images based on the detected differences between the captured images 31 acquired at different times. Therefore, the image recognition device 1 can select and store the detected differences 34 that occur consecutively, as shown in FIG.

[0055] (3) The transmission determination unit 14 has a spatial filter 142 that determines whether or not to store the detected difference based on the importance corresponding to the position of the detected difference in the photographed image 31. Therefore, the image recognition device 1 can store the detected difference 34 at an important position in the photographed image 31, as shown in FIG.

[0056] (4) The transmission determination unit 14 includes a time series filter 141 that calculates the occurrence of detected differences for multiple captured images 31 acquired at different times as a time series score, a spatial filter 142 that calculates importance according to the position of the detected difference in the captured image 31 as a position score, a score filter 143 that calculates a difference score 81 using the time series score and the position score, and a trigger determination unit 144 that determines whether or not to save the image based on the difference score 81. Therefore, the image recognition device 1 can select an appropriate captured image 31 from the perspectives of both time and space.

[0057] (5) The score filter 143 includes a score storage unit 202 that stores the difference scores 81, and the trigger determination unit 144 determines whether or not to store the difference scores 81 based on the difference scores 81 stored in the score storage unit 202. Therefore, the image recognition device 1 can select the captured image 31 that corresponds to the maximum value or the maximum value of the difference scores 81.

[0058] (6) The trigger determination unit 144 determines whether or not it is necessary to save at least one of the captured image 31 corresponding to the maximum difference score 81 within a specified period, the captured image 31 corresponding to the difference score 81 that exceeds a specified threshold, and the captured image 31 corresponding to the difference score 81 that is the maximum value.

[0059] (Variation 1) 11 is a configuration diagram of the image recognition device 1 in Modification 1. The image recognition device 1 includes an internal storage unit 18 that is a non-volatile storage device, and the trigger receiving unit 19 may store the captured image to be saved in the internal storage unit 18. In this case, the vehicle 9 does not need to include the external communication device 92. In this modification, the transmission determination unit 14 determines which captured image 31 to save in the image recognition device 1, and therefore the transmission determination unit 14 can be called a "save determination unit."

[0060] --Second embodiment-- A second embodiment of an image recognition device and a method for determining an image to be saved will be described with reference to Figures 12 and 13. In the following description, the same components as those in the first embodiment are designated by the same reference numerals, and differences will be mainly described. Points that are not particularly described are the same as those in the first embodiment. This embodiment differs from the first embodiment mainly in that only captured images that cannot be processed more appropriately by the new program than the current program are transmitted.

[0061] 12 is a configuration diagram of an image recognition device 1A according to the second embodiment. In addition to the configuration of the image recognition device 1 according to the first embodiment, the image recognition device 1A further includes a verification unit 15. The verification unit 15 receives a detected difference 34 from the difference extraction unit 13, calculates a reliability score 38, and outputs the reliability score 38 to the transmission determination unit 14. In addition to the processing described in the first embodiment, the transmission determination unit 14 also uses the reliability score 38 to output a transmission trigger 35.

[0062] The verification unit 15 verifies the deterioration in detection by the new program 12 relative to the current program 11 using a rule base or object detection, and calculates a reliability score 38. The reliability score 38 is a numerical value, for example, from 1 to 10, and the larger the value, the more likely it is that the detection difference 34 is caused by the deterioration of the new program 12 relative to the current program 11. For example, assume that the new program 12 is modified to newly detect object R relative to the current program 11, and the detection difference 34 occurs because only the new program 12 recognizes a certain area.

[0063] In this case, when the verification unit 15 performs object detection on the detection difference 34 and detects the object R, the detection difference 34 is the result of the new program 12 operating as intended, so it sets a low reliability score 38. However, if the result of performing object detection on the detection difference 34 does not detect any object, it means that the new program 12 has caused a false detection, so it sets a high reliability score 38.

[0064] 13 is a configuration diagram of a transmission determination unit 14A in the second embodiment. The transmission determination unit 14A includes a time series filter 141, a spatial filter 142, a score filter 143A, and a trigger determination unit 144. The time series filter 141 and the spatial filter 142 receive the detection difference 34 as in the first embodiment. The processing of the time series filter 141 and the spatial filter 142 is the same as in the first embodiment, so a description thereof will be omitted. The score filter 143A includes a performance degradation score calculation unit 301 and a score holding unit 202. The score filter 143A receives the time series filter output 36, the spatial filter output 37, the reliability score 38, and the new detection result 33.

[0065] The performance degradation score calculation unit 301 calculates the performance degradation score 82 as shown in the following equation 4, and outputs this performance degradation score 82 to the score storage unit 202. If the performance degradation score 82 at time t is represented as Sd(t), Sd(t) can be calculated as follows.

[0066] Sd(t)=Ct(t)P(t)Vc(t) / Dc(t) (Formula 4)

[0067] In Equation 4, Ct(t) is the time series filter output 36 at time t, P(t) is the spatial filter output 37 at time t, Vc(t) is the verification confidence score 38 at time t, and Dc(t) is the confidence of the detection result in the new program 12 at time t. In other words, the lower the confidence in determining performance degradation, the lower the performance degradation score Sd(t), and the lower the confidence Dc(t) of the detection result in the new program 12, the higher the performance degradation score Sd(t).

[0068] The trigger determination unit 144 generates a transmission trigger 35 by treating the time-series performance degradation score 82 stored in the score storage unit 202 in the same way as the difference score 81 in the first embodiment. That is, the trigger determination unit 144 generates a transmission trigger 35 for a captured image 31 corresponding to a performance degradation score 82 that is the maximum value for a predetermined period, a performance degradation score 82 that is a local maximum value, a performance degradation score 82 that is greater than a predetermined threshold, or the like.

[0069] According to the second embodiment described above, the following advantageous effects can be obtained. (7) The image recognition device 1A includes a verification unit 15 that outputs a reliability score 38 that indicates the degree to which the detected difference is caused by the deterioration of the new program 12 relative to the current program 11. The transmission determination unit 14 determines whether or not to save the detected difference 34 based on the occurrence status of the detected difference 34 and the reliability score 38. Therefore, the image recognition device 1A can select the detected difference 34 that has occurred because the new program 12 is inferior to the current program 11.

[0070] (8) The new program 12 outputs a new program reliability, which is the reliability of the detection result. The image recognition device 1A includes a time series filter 141 that calculates the appearance of detection differences in multiple captured images 31 acquired at different times as a time series filter output 36, also called a time series score, a spatial filter 142 that calculates the importance according to the position of the detection difference in the captured image 31 as a spatial filter output 37, also called a position score, and a performance degradation score calculation unit 301 that calculates a performance degradation score 82 based on the new program reliability, the time series filter output 36, the spatial filter output 37, and the reliability score 38. The transmission determination unit 14A determines whether or not storage is necessary based on the performance degradation score 82.

[0071] (9) The image recognition device 1A includes a score storage unit 202 that stores the performance degradation score 82. The trigger determination unit 144 of the transmission determination unit 14A determines whether or not the performance degradation score 82 needs to be stored based on the performance degradation score 82 stored in the score storage unit 202.

[0072] (Modification of the second embodiment) The performance degradation score calculation unit 301 may calculate the difference score S(t) as follows, without using the reliability score 38 calculated by the verification unit 15.

[0073] S(t)=Ct(t)P(t) / Dc(t) (Formula 5)

[0074] The subsequent processing and determination of this difference score S(t) is the same as in the second embodiment, and therefore a description thereof will be omitted.

[0075] -Third embodiment- A third embodiment of an image recognition device and a method for determining an image to be saved will be described with reference to Figures 14 and 15. In the following description, the same components as those in the first embodiment are designated by the same reference numerals, and differences will be mainly described. Points that are not particularly described are the same as those in the first embodiment. This embodiment differs from the first embodiment mainly in that the output of the control program in the subsequent stage is compared to determine the difference between the current program and the new program.

[0076] FIG. 14 is a configuration diagram of a vehicle 9B equipped with an image recognition device 1B according to the third embodiment. The vehicle 9B includes an image recognition device 1B, a camera 91, an exterior communication device 92, a control program 93, a second control program 93A, and a control difference extraction unit 96. The second control program 93A is identical to the control program 93, except that the output of the control program 93 is used to control the vehicle 9B, whereas the output of the second control program 93A is not used to control the vehicle 9B. The control difference extraction unit 96 extracts the difference between the output of the control program 93 and the output of the second control program 93A.

[0077] The control program 93 and the second control program 93A may be different instances generated from the same binary, or the same instance may be treated as pseudo-different instances in a time-sharing manner. However, the control program 93 and the second control program 93A do not need to be completely binary identical, as long as it is confirmed that differences in output are due to differences in input.

[0078] FIG. 15 is a diagram showing the relationship between the image recognition device 1B, the control program 93, the second control program 93A, and the control difference extraction unit 96. As in the first embodiment, the current program 11 outputs the current detection result 32 to the difference extraction unit 13 and the control program 93. The new program 12 outputs the new detection result 33 not only to the difference extraction unit 13 but also to the second control program 93A. The current detection result 32 is input to the control program 93, which outputs a first calculation result 94. The new detection result 33 is input to the second control program 93A, which outputs a second calculation result 94A. The control difference extraction unit 96 extracts the difference between the first calculation result 94 and the second calculation result 94A, and outputs a control difference 95 to the transmission determination unit 14.

[0079] The transmission determination unit 14 receives the detection difference 34 and the control difference 95. The transmission determination unit 14 may output a transmission trigger 35 based on these AND conditions. Alternatively, the transmission determination unit 14 may calculate a difference score 81 using not only the time series filter output 36 and the spatial filter output 37 but also the control difference 95, and generate a transmission trigger 35 for a captured image 31 corresponding to a maximum value, a local maximum value, or a value greater than a threshold value of the difference score 81.

[0080] According to the above-described third embodiment, the following advantageous effects can be obtained. (10) The current detection result 32, which is the object detection result of the current program 11, is used by the control program 93, which outputs the first calculation result 94. The first calculation result 94 is used to control the vehicle 9 equipped with the image recognition device 1. The transmission determination unit 14 determines whether or not to save the result based on a control difference 95, which is the difference between the first calculation result 94 and a second calculation result 94A obtained by inputting the new detection result 33, which is the object detection result of the new program 12, into the second control program 93A, and the appearance status of the detection difference 34. Therefore, by comparing the differences between the current program 11 and the new program 12 from the perspective of the output of the control program 93, which uses the output of the current program 11, it is possible to select captured images 31 that truly affect the entire system.

[0081] The first to third embodiments and their respective modifications described above may be implemented independently or in combination. Furthermore, the sensitivity to differential data collection and the upper limit of the frame rate of images that can be transmitted due to communication cost constraints vary depending on the system requirements and the verification phase. Therefore, it is possible to switch the configuration of each embodiment depending on the conditions. For example, immediately after updating the program and starting verification on an actual device, the configuration of the third embodiment may be used to extract truly important differential information to affect control, and then the configuration of the first or second embodiment may be used to adjust various parameters and collect differential data for conditions of interest.

[0082] Furthermore, the collected images 31 obtained in each embodiment may not only be transmitted or stored, but may also be used to improve the accuracy of the current detection results 32 used for vehicle control. For example, if there is a detection error in the current detection results 32 and the updated new detection results 33 are correct (if the performance of the updated version has been improved), and if the transmission determination unit determines that the accuracy is high, the verification results may be input to the subsequent control program 93, thereby realizing a driving system with higher performance.

[0083] In each of the above-described embodiments and modifications, the functional block configurations are merely examples. Some functional configurations shown as separate functional blocks may be configured as an integrated unit, or a configuration shown in a single functional block diagram may be divided into two or more functions. Furthermore, some of the functions of each functional block may be provided by other functional blocks.

[0084] In the above-described embodiments and modifications, the programs for implementing the difference extraction unit 13 and the transmission determination unit 14 are stored in the ROM 42. However, the programs may be stored in a non-volatile storage device. Furthermore, the image recognition device 1 may be provided with an input / output interface (not shown), and the programs may be loaded from another device as needed via the input / output interface and a medium available to the image recognition device 1. Here, the medium refers to, for example, a storage medium detachable from the input / output interface, or a communication medium, i.e., a wired, wireless, or optical network, or a carrier wave or digital signal propagating through the network. Furthermore, some or all of the functions implemented by the programs may be implemented by a hardware circuit or an FPGA.

[0085] The above-described embodiments and modifications may be combined with each other. Although various embodiments and modifications have been described above, the present invention is not limited to these. Other embodiments conceivable within the scope of the technical concept of the present invention are also included within the scope of the present invention. [Explanation of symbols]

[0086] 1, 1A, 1B: Image recognition device, 9, 9B: Vehicle, 11: Current program, 12: New program, 13: Difference extraction unit, 14, 14A: Transmission determination unit, 15: Verification unit, 18: Device memory unit, 31: Captured image, 32: Current detection result, 33: New detection result, 34: Detection difference, 35: Transmission trigger, 36: Time series filter output, 37: Spatial filter output, 38: Reliability score, 81: Difference score, 82: Performance degradation score, 9 1: camera, 93: control program, 93A: second control program, 94: first calculation result, 94A: second calculation result, 95: control difference, 96: control difference extraction unit, 141: time series filter, 142: spatial filter, 142M: risk determination map, 143: score filter, 143A: score filter, 144: trigger determination unit, 201: difference score calculation unit, 202: score storage unit, 301: performance degradation score calculation unit

Claims

1. An image recognition device that performs object detection on an input image using an image recognition program, a storage unit for storing a current program, which is the conventional image recognition program, and a new program, which is the new image recognition program; a difference extraction unit that extracts a detection difference that is a difference between an object detection result of the current program and an image recognition result of the new program for the same input image; a transmission determination unit that determines whether or not the input image needs to be saved based on the appearance status of the detected difference; An image recognition device comprising: a storage unit that outputs the input image that the transmission determination unit determines to be stored to an outside of the image recognition device, or stores the input image that the transmission determination unit determines to be stored in the image recognition device.

2. 2. The image recognition device according to claim 1, The transmission determination unit is an image recognition device having a time series filter that determines whether or not the image needs to be saved based on the detected differences for a plurality of the input images acquired at different times.

3. 2. The image recognition device according to claim 1, The transmission determination unit is an image recognition device having a spatial filter that determines whether or not the detected difference needs to be saved based on the importance according to the position of the detected difference in the input image.

4. 2. The image recognition device according to claim 1, The transmission determination unit a time-series filter that calculates the occurrence of the detected difference for the plurality of input images acquired at different times as a time-series score; a spatial filter that calculates a position score based on the importance of the detected difference in the input image; a score filter that calculates a difference score using the time series score and the position score; a trigger determination unit that determines whether or not the image needs to be saved based on the difference score.

5. 5. The image recognition device according to claim 4, the score filter further includes a score storage unit for storing the difference score; The trigger determination unit determines whether or not the storage is necessary based on the difference score stored in the score storage unit.

6. 6. The image recognition device according to claim 5, The trigger determination unit determines whether or not to store at least one of the input image corresponding to the maximum difference score within a predetermined period, the input image corresponding to the difference score that exceeds a predetermined threshold, and the input image corresponding to the difference score that is a maximum value.

7. 2. The image recognition device according to claim 1, a verification unit that outputs a reliability score indicating the degree to which the detected difference is due to a deterioration of the new program relative to the current program; The transmission determination unit determines whether or not the image needs to be saved based on the appearance status of the detected difference and the reliability score.

8. 8. The image recognition device according to claim 7, The new program further outputs a new program reliability, which is a reliability of the detection result; a time-series filter that calculates the occurrence of the detected difference for the plurality of input images acquired at different times as a time-series score; a spatial filter that calculates a position score based on the importance of the detected difference in the input image; a performance degradation score calculation unit that calculates a performance degradation score based on the new program reliability, the time series score, the position score, and the reliability score, The transmission determination unit is an image recognition device that determines whether or not the image needs to be saved based on the performance degradation score.

9. 9. The image recognition device according to claim 8, a score storage unit for storing the performance degradation score, The transmission determination unit determines whether or not the storage is necessary based on the performance degradation score stored in the score storage unit.

10. 2. The image recognition device according to claim 1, The object detection result of the current program is used in a first program that outputs a first calculation result; the first calculation result is used to control a vehicle equipped with the image recognition device, The transmission determination unit determines whether or not the storage is necessary based on a control difference, which is the difference between a second calculation result obtained by inputting the object detection result of the new program into the first program and the first calculation result, and the appearance status of the detection difference.

11. 8. The image recognition device according to claim 7, The object detection result of the current program is used in a first program that outputs a first calculation result; the first calculation result is used to control a vehicle equipped with the image recognition device, The transmission determination unit determines whether or not the storage is necessary based on a control difference, which is the difference between a second calculation result obtained by inputting the object detection result of the new program into the first program and the first calculation result, and the appearance status of the detection difference.

12. A method for determining an image to be saved, which is executed by an image recognition device that performs object detection on an input image using an image recognition program, comprising: the image recognition device includes a storage unit that stores a current program that is the conventional image recognition program and a new program that is the new image recognition program, a difference extraction step of extracting a detection difference, which is a difference between the object detection result of the current program and the image recognition result of the new program for the same input image; a transmission determination step of determining whether or not the input image needs to be saved based on the appearance status of the detected difference; a storing step of outputting the input image determined to be saved by the transmission determination step to an outside of the image recognition device, or storing the input image determined to be saved by the transmission determination step in the image recognition device.

Images