Sensing information recognition system

Abstract

The purpose of the present invention is to improve the reliability of recognition results while minimizing a hardware increase. This sensing information recognition system recognizes sensing information obtained from a sensor installed in a vehicle, said system comprising: a first processing device that executes first recognition processing on the sensing information; and a second processing device that executes second recognition processing on the sensing information. The second processing device executes the second recognition processing for a longer duration than the duration for which the first processing device executes the first recognition processing. The first processing device assesses the reliability of results of the first recognition processing on the basis of results of the second recognition processing executed by the second processing device.

Description

Sensing Information Recognition System

[0001] The present invention relates to a sensing information recognition system.

[0002] A sensing information recognition system is known that recognizes sensing information acquired from sensors mounted on a vehicle. In the sensing information recognition system, high reliability is required for the recognition results of the sensing information. To increase the reliability of the recognition results, it is conceivable to perform recognition processing using multiple inference models constructed by large-scale artificial intelligence (AI).

[0003] For example, Patent Document 1 describes an image recognition device that recognizes images captured by a camera device mounted on a vehicle. The image recognition device described in Patent Document 1 executes image recognition processing using multiple image recognition models and integrates the results of the multiple image recognition processing to obtain a highly reliable recognition result.

[0004] JP 2023-113170 A

[0005] However, the technology described in Patent Document 1 requires a large amount of calculation to increase the reliability of the recognition results, which necessitates strengthening the hardware, resulting in increased costs and power resources.

[0006] The present invention has been made in view of the above, and has as its object to improve the reliability of recognition results while minimizing the need for increased hardware.

[0007] In order to solve the above problem, the sensing information recognition system of the present invention is a sensing information recognition system that recognizes sensing information acquired from a sensor mounted on a vehicle, and is equipped with a first processing device that performs a first recognition process on the sensing information, and a second processing device that performs a second recognition process on the sensing information, wherein the second processing device performs the second recognition process for a longer time than the time it takes for the first processing device to perform the first recognition process, and the first processing device determines the reliability of the result of the first recognition process based on the result of the second recognition process performed by the second processing device.

[0008] According to the present invention, it is possible to improve the reliability of the recognition results while minimizing the increase in hardware. Problems, components, and effects other than those described above will become apparent from the following description of the embodiments.

[0009] 1 is a block diagram illustrating the configuration of a sensing information recognition system according to a first embodiment; a flowchart illustrating processing executed by the first processing device and the second processing device shown in FIG. 1; a diagram illustrating the difference between the first embodiment and the prior art; a block diagram illustrating the configuration of a sensing information recognition system according to a second embodiment; a flowchart illustrating processing executed by the first processing device and the second processing device shown in FIG. 4; a diagram illustrating the second embodiment and corresponding to FIG. 3; a block diagram illustrating the configuration of a sensing information recognition system according to a third embodiment; a flowchart illustrating processing executed by the first processing device and the second processing device shown in FIG. 7; a diagram illustrating the third embodiment and corresponding to FIG. 3; a block diagram illustrating the configuration of a sensing information recognition system according to a fourth embodiment; a flowchart illustrating processing executed by the first processing device and the second processing device shown in FIG. 10; a diagram illustrating the fourth embodiment and corresponding to FIG. 3.

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that components with the same reference numerals in each embodiment have similar components in each embodiment unless otherwise specified, and description thereof will be omitted.

[0011] First Embodiment A first embodiment of the present invention will be described with reference to FIGS. 1 to 3. FIG.

[0012] The sensing information recognition system 10 is a system that recognizes sensing information acquired from a sensor mounted on the vehicle 1. The sensing information acquired from the sensor mounted on the vehicle 1 is, for example, an image acquired by an image sensor 3 provided in a camera device 2 mounted on the vehicle 1. Alternatively, the sensing information acquired from the sensor mounted on the vehicle 1 is, for example, information acquired by various sensors mounted on the vehicle 1, such as an infrared sensor, a millimeter wave sensor, a sound sensor, a pressure sensor, an acceleration sensor, or a sensor that detects changes in capacitance. In the following, the image sensor 3 will be used as an example of the sensor mounted on the vehicle 1, and an image acquired by the image sensor 3 will be used as an example of the sensing information.

[0013] FIG. 1 is a block diagram illustrating the configuration of a sensing information recognition system 10 according to the first embodiment.

[0014] 1 , the camera device 2 mounted on the vehicle 1 includes an image sensor 3 that converts light collected by a lens (not shown) into an electrical signal to acquire an image, and an image recognition device 11 that performs recognition processing on the image acquired by the image sensor 3. The image recognition device 11 is connected to a vehicle bus 4, which is an in-vehicle network of the vehicle 1. The sensing information recognition system 10 of the first embodiment is configured with the image recognition device 11.

[0015] The image recognition device 11 includes a first processing device 12 , a second processing device 13 , a memory 14 , an ISP (Image Signal Processor) 15 , and a vehicle bus I / F 16 , which are connected by an internal bus 17 .

[0016] The ISP 15 performs image correction processes such as distortion correction, exposure adjustment, and color correction on the image acquired by the image sensor 3, and stores the image in the memory 14 as an input image 21. The ISP 15 performs a series of processes, namely, acquiring the image from the image sensor 3, performing the image correction process, and storing the image in the memory 14, at a preset frame cycle (1 / fps).

[0017] The numbers (1) to (9) attached to the arrows shown in Fig. 1 indicate the order in which data, information, signals, etc. are transferred. The same numbers indicate that data, etc. are transferred in the same frame period. This also applies to Figs. 4, 7, and 10.

[0018] The first processing device 12 is configured with a processor and performs recognition processing on the sensing information. In this embodiment, the recognition processing performed by the first processing device 12 is also referred to as a "first recognition processing." In this embodiment, the first processing device 12 performs processing to detect an object included in an input image 21 as the first recognition processing. After performing the first recognition processing, the first processing device 12 stores a first recognition result 22, which is the result of the first recognition processing, in the memory 14.

[0019] The first processing device 12 executes the first recognition process using a first inference model constructed by AI. The first processing device 12 of this embodiment executes the first recognition process by inputting an input image 21 stored in the memory 14 into the first inference model. The first inference model is an inference model that has undergone machine learning. For example, the first inference model of this embodiment is a model that makes extensive use of CNN (Convolutional Neural Network) operations such as ResNet (Residual Network) and skip connections. ResNet is an algorithm that has a relatively low computational load (requiring a small number of integer operations) through adjustment of the layer structure, etc., yet is capable of performing accurate object classification (classifying object types).

[0020] Furthermore, the first processing device 12 of this embodiment uses the first recognition result 22 to execute predetermined processing necessary for controlling an actuator that moves the vehicle 1. The actuator that moves the vehicle 1 is, for example, an accelerator device that accelerates the vehicle 1, a brake device that decelerates the vehicle 1, or a steering device that steers the vehicle 1.

[0021] The predetermined processing required for controlling the actuators that move the vehicle 1 includes an object distance estimation processing that estimates the distance to an object detected by the first recognition processing. The predetermined processing includes an object tracing processing that traces the object detected by the first recognition processing over several frames of input images 21 and calculates the velocity vector of the object. The first processing device 12 of this embodiment stores an object distance estimation result 23 that is the result of the object distance estimation processing and an object tracing result 24 that is the result of the object tracing processing in the memory 14.

[0022] Furthermore, the predetermined processing required to control the actuator that moves the vehicle 1 includes processing to compare the velocity vector of the object detected by the first recognition processing with the velocity vector of the vehicle 1 and generate a control command 25 for controlling the actuator. The first processing device 12 of this embodiment stores the generated control command 25 in the memory 14.

[0023] The second processing device 13 is configured with a processor and performs recognition processing on the sensing information. In this embodiment, the recognition processing performed by the second processing device 13 is also referred to as "second recognition processing." As the second recognition processing, the second processing device 13 of this embodiment performs processing to detect objects included in the input image 21, similar to the first recognition processing. After performing the second recognition processing, the second processing device 13 stores a second recognition result 26, which is the result of the second recognition processing, in the memory 14.

[0024] The second processing device 13 executes the second recognition processing using a second inference model constructed by AI. The second processing device 13 of this embodiment executes the second recognition processing by inputting the input image 21 stored in the memory 14 into the second inference model. That is, the second processing device 13 executes the second recognition processing by inputting the same input information as the first inference model into the second inference model. The second inference model is an inference model that has undergone machine learning. The second inference model has an algorithm different from that of the first inference model. For example, the second inference model of this embodiment is a model that utilizes an attention mechanism or a fully connected layer in addition to CNN operations such as the Vision Transformer. The second inference model is capable of extracting features different from those of a CNN-based model while still performing similar object classification.

[0025] The second inference model has a greater computational load (requiring a greater number of integer operations) than the first inference model, but the reliability of the inference result is higher. That is, the second processing device 13 executes the second recognition process, which has a greater computational load than the first recognition process executed by the first processing device 12, but obtains the second recognition result 26, which is more reliable than the first recognition result 22. For example, the accuracy rate of the first recognition result 22 is about 90%, and the number of object classes in the first recognition result 22 is about 3 to 5. On the other hand, the accuracy rate of the second recognition result 26 is 99% or higher, and the number of object classes in the second recognition result 26 is 20 or more.

[0026] However, the second processing device 13 executes the second recognition process for a longer time than the time it takes for the first processing device 12 to execute the first recognition process. In other words, the first processing device 12 executes the first recognition process for a shorter time than the time it takes for the second processing device 13 to execute the second recognition process. This allows the second processing device 13 to obtain a second recognition result 26 that is more reliable than the first recognition result 22 without having a hardware processing capability that is higher than that of the first processing device 12. The hardware processing capability of the second processing device 13 may be equivalent to the hardware processing capability of the first processing device 12. For example, the first processing device 12 and the second processing device 13 may be configured by processors having the same TOPs (Tera Operations Per second).

[0027] Furthermore, the first processing device 12 determines the reliability of the first recognition result 22, which is the result of the first recognition processing, based on the second recognition result 26, which is the result of the second recognition processing executed by the second processing device 13. In other words, the first processing device 12 determines the reliability of the first recognition result 22 based on the second recognition result 26, which is more reliable than the first recognition result 22.

[0028] Specifically, the first processing device 12 acquires the first recognition result 22 and the second recognition result 26 stored in the memory 14. The first processing device 12 then determines whether the first recognition result 22 is consistent with the second recognition result 26, thereby determining the reliability of the first recognition result 22. If the first recognition result 22 is consistent with the second recognition result 26, the first processing device 12 determines that the first recognition result 22 is reliable. If the first recognition result 22 is inconsistent with the second recognition result 26, the first processing device 12 determines that the first recognition result 22 is unreliable. The reliability of the first recognition result 22 means that the first recognition result 22 is an object detection result that is accurate enough to ensure the safety of the vehicle 1 even if the actuator that moves the vehicle 1 is controlled based on the first recognition result 22. The first processing device 12 of this embodiment stores a determination result 27 of whether the first recognition result 22 is reliable in the memory 14.

[0029] Then, when the first processing device 12 determines that the first recognition result 22 is reliable, it transmits the result of the above-mentioned predetermined processing to the control device 9 (see FIG. 7 ) that controls the actuator. More specifically, the first processing device 12 of this embodiment outputs a control command 25, which is one of the results of the above-mentioned predetermined processing and is stored in the memory 14, to the vehicle bus I / F 16. As a result, the control command 25 is transmitted to the control device 9 via the vehicle bus 4. Note that the first processing device 12 may transmit not only the control command 25 but also the object distance estimation result 23 and the object tracing result 24 to the control device 9.

[0030] On the other hand, if the first processing device 12 determines that the first recognition result 22 is not reliable, it discards the results of the above-mentioned predetermined processing. More specifically, the first processing device 12 of this embodiment discards the object distance estimation result 23, the object tracing result 24, and the control command 25 stored in the memory 14. Furthermore, in this case, the first processing device 12 also discards the first recognition result 22 stored in the memory 14.

[0031] FIG. 2 is a flowchart showing the processing executed by the first processor 12 and the second processor 13 shown in FIG.

[0032] The first processing device 12 acquires the input image 21 stored in the memory 14 (step S11). Then, the first processing device 12 executes a first recognition process using the first inference model (step S12) and stores the first recognition result 22 in the memory 14. At the same time, the second processing device 13 acquires the input image 21 stored in the memory 14 (step S21) and executes a second recognition process using the second inference model (step S22).

[0033] The first processing device 12 executes the above-described predetermined processing during the period from when the first recognition processing is executed until when the second recognition result 26 is acquired (between steps S12 and S16). Specifically, the first processing device 12 executes an object distance estimation processing (step S13) and stores an object distance estimation result 23 in the memory 14. The first processing device 12 executes an object tracing processing (step S14) and stores an object tracing result 24 in the memory 14. The first processing device 12 generates a control command 25 (step S15) and stores it in the memory 14. During steps S12 to S16, the second processing device 13 is executing the second recognition processing. When the second recognition result 26 is acquired by executing the second recognition processing, the second processing device 13 stores the second recognition result 26 in the memory 14.

[0034] When the second recognition result 26 is stored in the memory 14, the first processing device 12 acquires the first recognition result 22 and the second recognition result 26 stored in the memory 14. The first processing device 12 then determines the reliability of the first recognition result 22 based on the acquired second recognition result 26. Specifically, the first processing device 12 determines whether the first recognition result 22 is consistent with the second recognition result 26 (step S16). If the first recognition result 22 is consistent with the second recognition result 26 (step S16: Yes), the first processing device 12 transmits the control command 25 stored in the memory 14 as a result of the predetermined processing to the control device 9 (step S17). If the first recognition result 22 is inconsistent with the second recognition result 26 (step S16: No), the first processing device 12 discards the object distance estimation result 23, the object tracing result 24, and the control command 25 stored in the memory 14 as a result of the predetermined processing (step S18). Furthermore, in this case, the first processing device 12 also discards the first recognition result 22 stored in the memory 14 (step S18). After that, the first processing device 12 and the second processing device 13 end the processing shown in FIG.

[0035] 3 is a diagram illustrating the difference between the first embodiment and the prior art, showing the processing executed by the first processing device (12) or the second processing device (13) for each frame period.

[0036] As described above, the sensing information recognition system 10 of the first embodiment is a sensing information recognition system that recognizes sensing information acquired from a sensor mounted on the vehicle 1, and includes a first processing device 12 that executes a first recognition process on the sensing information, and a second processing device 13 that executes a second recognition process on the sensing information. The second processing device 13 executes the second recognition process for a longer time than the time it takes for the first processing device 12 to execute the first recognition process. The first processing device 12 determines the reliability of a first recognition result 22 that is the result of the first recognition process, based on a second recognition result 26 that is the result of the second recognition process executed by the second processing device 13.

[0037] As a result, the second processing device 13 can execute the second recognition process, which has a greater computational load than the first recognition process, without having a higher processing power as hardware than the first processing device 12, and can obtain a second recognition result 26 that is more reliable than the first recognition result 22. Even if a less reliable first recognition result 22 is obtained by executing the first recognition process, which has a smaller computational load, the first processing device 12 can determine the reliability of the first recognition result 22 based on the more reliable second recognition result 26, and can therefore adopt only the more reliable first recognition result 22. Thus, the sensing information recognition system 10 can improve the reliability of the recognition results while minimizing hardware enhancements.

[0038] Consider a case where the second processing device executes the second recognition process in the same time as the first processing device executes the first recognition process, as in the "prior art" shown in Figure 3. In this case, the second processing device needs to perform the second recognition process, which has a greater computational load than the first recognition process, in order to obtain a second recognition result that is more reliable than the first recognition result, and therefore the processing power of the second processing device needs to be increased.

[0039] Furthermore, when the first processing device integrates the first recognition result and the second recognition result without determining the reliability of the first recognition result based on the second recognition result, the reliability of the first recognition result needs to be made as high as that of the second recognition result. To achieve this, the first processing device needs to either extend the execution time of the first recognition process or increase the processing power of the first processing device in order to execute the first recognition process, which has a large computational load like the second recognition process.

[0040] If the execution time of the first recognition process is extended, the timing of executing the object distance estimation process, object tracing process, and control command generation, which are executed using the first recognition result, will be delayed. This will delay the timing of executing the control of the actuators described above, and there is a possibility that the vehicle 1 will not be able to move with sufficient time after sensing. Therefore, in order to execute the first recognition process, which has a large calculation load like the second recognition process, the only option is to increase the processing capacity of the first processing device.

[0041] That is, in the prior art, in order to improve the reliability of the recognition results, it is necessary to increase the processing power of the first processing device or the second processing device, which inevitably requires an increase in hardware.

[0042] In contrast, as described above, the sensing information recognition system 10 of the first embodiment can execute the object distance estimation process, the object tracing process, the generation of control commands, and the like without delay by using only the highly reliable first recognition result 22, without increasing the processing power of the first processing device 12 or the second processing device 13. Therefore, the sensing information recognition system 10 does not need to increase the processing power of the first processing device 12 or the second processing device 13 in order to improve the reliability of the recognition result. Therefore, the sensing information recognition system 10 can improve the reliability of the recognition result while minimizing the need to increase the hardware.

[0043] Furthermore, in the sensing information recognition system 10 of the first embodiment, the first processing device 12 executes a first recognition process using a first inference model, and the second processing device 13 executes a second recognition process using a second inference model. The second processing device 13 inputs the same input information as the first inference model of the first processing device 12 into the second inference model and executes the second recognition process.

[0044] This enables the sensing information recognition system 10 to further increase the reliability of each of the first recognition result 22 and the second recognition result 26. The sensing information recognition system 10 can then reliably determine the reliability of the first recognition result 22 based on the highly reliable second recognition result 26, and can reliably adopt only the highly reliable first recognition result 22. Thus, the sensing information recognition system 10 can further improve the reliability of the recognition results while minimizing hardware enhancements.

[0045] Furthermore, in the sensing information recognition system 10 of the first embodiment, the first processing device 12 executes a predetermined process required to control the actuator that moves the vehicle 1, using the first recognition result 22, between the time when the first processing device 12 executes the first recognition process and the time when the first processing device 12 acquires the second recognition result 26, which is the result of the second recognition process executed by the second processing device 13.

[0046] This allows the sensing information recognition system 10 to execute the aforementioned predetermined processes, such as the object distance estimation process, the object tracing process, and the generation of control commands, without delay. Therefore, the sensing information recognition system 10 does not need to increase the processing power of the first processing device 12 or the second processing device 13 in order to improve the reliability of the recognition results. Therefore, the sensing information recognition system 10 can improve the reliability of the recognition results while minimizing the need to increase the hardware.

[0047] Furthermore, in the sensing information recognition system 10 of the first embodiment, the first processing device 12 determines reliability by determining whether the first recognition result 22 is consistent with the second recognition result 26. If the first recognition result 22 is consistent with the second recognition result 26, the first processing device 12 transmits the result of the predetermined processing described above to the control device 9. If the first recognition result 22 is not consistent with the second recognition result 26, the first processing device 12 discards the result of the predetermined processing.

[0048] This allows the sensing information recognition system 10 to transmit only the results of the predetermined processing executed using the first recognition result 22 that has been determined to be highly reliable to the control device 9. The control device 9 can more accurately control the actuators that move the vehicle 1. Therefore, the sensing information recognition system 10 can improve the reliability of the recognition results and the accuracy of actuator control while minimizing the increase in hardware.

[0049] Second Embodiment A second embodiment of the present invention will be described with reference to Figures 4 to 6. In the second embodiment, the description of the same components as those in the first embodiment will be omitted.

[0050] Fig. 4 is a block diagram illustrating the configuration of a sensing information recognition system 10 according to a second embodiment. Fig. 5 is a flowchart illustrating processing executed by the first processing device 12 and the second processing device 13 shown in Fig. 4. Fig. 6 is a diagram illustrating the second embodiment and corresponds to Fig. 3.

[0051] The second processing device 13 of the first embodiment inputs the same input information as the first inference model of the first processing device 12 into the second inference model and executes the second recognition process. In contrast, the second processing device 13 of the second embodiment inputs the same input information as the first inference model of the first processing device 12 and multiple pieces of additional information related to the input information into the second inference model and executes the second recognition process.

[0052] In the second embodiment, the input information identical to the first inference model of the first processing device 12 is the input image 21a stored in the memory 14 by the ISP 15, as shown in Figure 4. The multiple pieces of additional information connected to the input information are the input images 21b and 21c stored in the memory 14 consecutively to the input image 21a by the ISP 15. The input images 21a to 21c are images of consecutive frames.

[0053] That is, as shown in Fig. 5, the first processing device 12 of the second embodiment acquires an input image 21a stored in the memory 14 (step S31), inputs the input image 21a to a first inference model, and executes a first recognition process (step S32). On the other hand, the second processing device 13 of the second embodiment acquires multiple frames of input images 21a to 21c stored in the memory 14 (step S41), inputs the multiple frames of input images 21a to 21c to a second inference model, and executes a second recognition process (step S42). The multiple frames of input images 21a to 21c are images acquired in multiple consecutive frame periods, as shown in Fig. 6.

[0054] The first processing device 12 of the second embodiment not only determines whether the first recognition result 22 is consistent with the second recognition result 26 for the input image 21a, but also determines whether the first recognition result 22 is consistent with the second recognition results 26 for the multiple frames of input images 21a to 21c (step S36). Specifically, the first processing device 12 not only determines whether the object to be determined included in the first recognition result 22 matches the object included in the second recognition result 26 for the input image 21a, but also determines whether the object is moving as expected in the second recognition results 26 for the input images 21b and 21c. For example, if the object to be determined included in the first recognition result 22 is a pedestrian, the first processing device 12 determines whether the pedestrian is moving as expected in the second recognition results 26 for the input images 21b and 21c. If the object to be determined included in the first recognition result 22 matches the object included in the second recognition result 26 for the input image 21 a and is moving as expected (step S36: Yes), the first processing device 12 determines that the first recognition result 22 is consistent with the second recognition result 26 and is reliable. If the object to be determined included in the first recognition result 22 does not match the object included in the second recognition result 26 for the input image 21 a or is not moving as expected (step S36: No), the first processing device 12 determines that the first recognition result 22 is not consistent with the second recognition result 26 and is unreliable.

[0055] As a result, the sensing information recognition system 10 of the second embodiment can further increase the reliability of the second recognition result 26. The sensing information recognition system 10 of the second embodiment can more reliably determine the reliability of the first recognition result 22 based on the highly reliable second recognition result 26, and can more reliably adopt only the highly reliable first recognition result 22. Therefore, the sensing information recognition system 10 of the second embodiment can further improve the reliability of the recognition result while minimizing the increase in hardware.

[0056] The second processing device 13 of the second embodiment may input not only the input images 21a to 21c of multiple frames but also the object distance estimation result 23 and the object tracing result 24 to the second inference model (steps S33 and S34) to execute the second recognition process. That is, the multiple pieces of additional information input to the second inference model of the second processing device 13 may include not only the input images 21b and 21c but also the object distance estimation result 23 and the object tracing result 24. This allows the sensing information recognition system 10 of the second embodiment to further increase the reliability of the second recognition result 26.

[0057] Third Embodiment A third embodiment of the present invention will be described with reference to Figures 7 to 9. In the third embodiment, the description of the same components as those in the first embodiment will be omitted.

[0058] Fig. 7 is a block diagram illustrating the configuration of a sensing information recognition system 10 according to a third embodiment. Fig. 8 is a flowchart illustrating processing executed by the first processing device 12 and the second processing device 13 shown in Fig. 7. Fig. 9 is a diagram illustrating the third embodiment and corresponds to Fig. 3.

[0059] The sensing information recognition system 10 of the first embodiment is composed of an image recognition device 11 included in a camera device 2 mounted on a vehicle 1, and both a first processing device 12 and a second processing device 13 are provided in the image recognition device 11.

[0060] In contrast, as shown in FIG. 7 , a sensing information recognition system 10 of the third embodiment includes an image recognition device 11 included in a camera device 2 mounted on a vehicle 1, and a server 8 connected to the vehicle 1 via a communication network 7. The image recognition device 11 of the third embodiment is connected to a network I / F 6 via a vehicle bus 4 and a vehicle bus I / F 5, and is connected to the server 8 via the network I / F 6 and the communication network 7. The image recognition device 11 of the third embodiment is also connected to a control device 9 via the vehicle bus 4 and the vehicle bus I / F 5. The first processing device 12 of the third embodiment is mounted on the image recognition device 11 of the vehicle 1. The second processing device 13 of the third embodiment is mounted on the server 8. The server 8 may be a cloud server, and can collect map information and weather information around the vehicle 1, as well as road information and other vehicle information obtained through vehicle-to-vehicle communication or road-to-vehicle communication.

[0061] 8 and 9 , the second processing device 13 of the third embodiment acquires the input image 21, which is input information transmitted from the vehicle 1 to the server 8 via the communication network 7 (step S61). Then, the second processing device 13 of the third embodiment inputs the acquired input information, which is the input image 21, into a second inference model and executes a second recognition process (step S62).

[0062] Here, since the second processing device 13 of the third embodiment is mounted on the server 8, it has a higher-performance computing function than when mounted on the vehicle 1, and can obtain a more reliable second recognition result 26 than when mounted on the vehicle 1. For example, the second processing device 13 of the third embodiment can additionally learn the second inference model in cooperation with at least one of map information, weather information, and information obtained through vehicle-to-vehicle communication or road-to-vehicle communication collected by the server 8, and update the second inference model so that it has the latest learning parameters. The second processing device 13 of the third embodiment can then perform the second recognition process using the latest second inference model. The second processing device 13 of the third embodiment transmits the second recognition result 26 from the server 8 to the vehicle 1 via the communication network 7 (step S62). The second recognition result 26 is stored in the memory 14 by the vehicle bus I / F 16. The first processing device 12 of the third embodiment determines the reliability of the first recognition result 22 based on the second recognition result 26 transmitted from the server 8 to the vehicle 1 via the communication network 7 (step S56).

[0063] As a result, the sensing information recognition system 10 of the third embodiment can further increase the reliability of the second recognition result 26. The sensing information recognition system 10 of the third embodiment can more reliably determine the reliability of the first recognition result 22 based on the highly reliable second recognition result 26, and can more reliably adopt only the highly reliable first recognition result 22. Therefore, the sensing information recognition system 10 of the third embodiment can further improve the reliability of the recognition result while minimizing the increase in hardware.

[0064] Fourth Embodiment A fourth embodiment of the present invention will be described with reference to Figures 10 to 12. In the fourth embodiment, the description of the same components as those in the first embodiment will be omitted.

[0065] Fig. 10 is a block diagram illustrating the configuration of a sensing information recognition system 10 according to a fourth embodiment. Fig. 11 is a flowchart illustrating processing executed by the first processing device 12 and the second processing device 13 shown in Fig. 10. Fig. 12 is a diagram illustrating the fourth embodiment and corresponds to Fig. 3.

[0066] The camera device 2 of the first embodiment is a camera device equipped with one image sensor 3, and the image recognition device 11 of the first embodiment performs recognition processing on an input image 21 acquired by the one image sensor 3. Specifically, the first processing device 12 and the second processing device 13 of the first embodiment input the input image 21 to a first inference model and a second inference model, respectively, and perform a first recognition processing and a second recognition processing.

[0067] In contrast, the camera device 2 of the fourth embodiment is a stereo camera device equipped with a pair of image sensors 3, as shown in FIG. 10 , and the image recognition device 11 of the fourth embodiment performs recognition processing on stereo images 21d, 21e acquired by the pair of image sensors 3.

[0068] 11, the first processing device 12 of the fourth embodiment acquires stereo images 21d and 21e as the input image 21 (step S71). Then, as shown in FIGS. 11 and 12, the first processing device 12 of the fourth embodiment executes stereo matching processing on the stereo images 21d and 21e as the first recognition processing (step S72).

[0069] The stereo matching process generates a parallax image from the stereo images 21d and 21e and detects the presence area of ​​an arbitrary object contained in the parallax image. An arbitrary object is an object whose type has not yet been identified. Since distance information to the arbitrary object is added to each pixel constituting the parallax image, the first processing device 12 of the fourth embodiment can obtain an object distance estimation result 23 for the arbitrary object by generating the parallax image and can detect the presence of the arbitrary object in a specific area on the parallax image. The first processing device 12 of the fourth embodiment stores an arbitrary object area image 28 indicating the presence area of ​​the detected arbitrary object and the object distance estimation result 23 in the memory 14 as a first recognition result 22.

[0070] The second processing device 13 of the fourth embodiment acquires the arbitrary object area image 28 stored in the memory 14 (step S81). Then, the second processing device 13 of the fourth embodiment inputs the arbitrary object area image 28 into an inference model related to the identification of the arbitrary object and executes the second recognition process (step S82). As a result, the second processing device 13 of the fourth embodiment can identify the type of the arbitrary object included in the arbitrary object area image 28, and stores the obtained identification result of the arbitrary object in the memory 14 as the second recognition result 26. The inference model used by the second processing device 13 of the fourth embodiment is a machine-learned inference model that identifies the type of the arbitrary object from the arbitrary object area image 28. This inference model has a smaller computational load than the second inference model of the first embodiment.

[0071] The first processing device 12 of the fourth embodiment determines the reliability of the first recognition result 22 based on the object identification result, which is the second recognition result 26. Specifically, the first processing device 12 of the fourth embodiment identifies the type of object present in the presence area detected by the first recognition process from the object identification result, which is the second recognition result 26. Then, the first processing device 12 of the fourth embodiment determines, from the identified type of object, whether the object present in the presence area detected by the first recognition process is a reliable object for controlling an actuator that moves the vehicle 1 (step S75). If the object present in the presence area detected by the first recognition process is a reliable object for controlling the actuator (step S75: Yes), the first processing device 12 of the fourth embodiment transmits the control command 25 stored in the memory 14 as a result of the predetermined process described above to the control device 9 (step S76). If the arbitrary object present in the presence area detected by the first recognition processing is not a reliable object for controlling the actuator (step S75: No), the first processing device 12 of the fourth embodiment discards the object tracing result 24 and the control command 25 stored in the memory 14 as the results of the above-mentioned predetermined processing (step S77). Furthermore, in this case, the first processing device 12 of the fourth embodiment also discards the object distance estimation result 23 and the arbitrary object area image 28 stored in the memory 14 as the first recognition result 22 (step S77).

[0072] As a result, the sensing information recognition system 10 of the fourth embodiment can transmit to the control device 9 only the results of the above-mentioned predetermined processing that has been executed using the first recognition results 22 that have been determined to be highly reliable by the second recognition results 26, out of the first recognition results 22 obtained as a result of the stereo matching processing. The control device 9 can more accurately control the actuators that move the vehicle 1. Therefore, the sensing information recognition system 10 of the fourth embodiment can improve the reliability of the recognition results and the accuracy of actuator control while minimizing the increase in hardware.

[0073] The present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those including all of the described components. Furthermore, some of the components of one embodiment can be replaced with components of another embodiment, and components of another embodiment can be added to components of one embodiment. Furthermore, some of the components of each embodiment can be added, deleted, or replaced with other components.

[0074] Furthermore, the above-described components, functions, processing units, or processing means may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described components or functions may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the program, table, or file that implements each function can be stored in a storage device such as a memory, a hard disk, or an SSD (solid state drive), or in a storage medium such as an IC card, an SD card, or a DVD.

[0075] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected.

[0076] 1...vehicle, 2...camera device, 3...image sensor, 7...communication network, 8...server, 9...control device, 10...sensing information recognition system, 12...first processing device, 13...second processing device, 21...input image, 21a to 21c...multiple frames of input images (additional information), 21d, 21e...stereo images, 22...first recognition result, 23...object distance estimation result (result of predetermined processing), 24...object tracing result (result of predetermined processing), 25...control command (result of predetermined processing), 26...second recognition result, 28...arbitrary object area image

Claims

1. A sensing information recognition system that recognizes sensing information obtained from a sensor mounted on a vehicle, comprising: a first processing device that performs a first recognition process on the sensing information; and a second processing device that performs a second recognition process on the sensing information, wherein the second processing device performs the second recognition process for a longer time than the time it takes for the first processing device to perform the first recognition process, and the first processing device determines the reliability of the result of the first recognition process based on the result of the second recognition process performed by the second processing device.

2. The sensing information recognition system described in claim 1, characterized in that the first processing device performs the first recognition process using a first inference model, the second processing device performs the second recognition process using a second inference model, and the second processing device inputs the same input information as the first inference model into the second inference model to perform the second recognition process.

3. The sensing information recognition system described in claim 2, characterized in that the first processing device is mounted on the vehicle, the second processing device is mounted on a server connected to the vehicle via a communication network, the second processing device inputs the input information transmitted from the vehicle to the server via the communication network into the second inference model to perform the second recognition processing, and transmits the result of the second recognition processing from the server to the vehicle via the communication network, and the first processing device determines the reliability of the result of the first recognition processing based on the result of the second recognition processing transmitted from the server to the vehicle via the communication network.

4. The sensing information recognition system described in claim 2, characterized in that the second processing device inputs the same input information as the first inference model and multiple additional pieces of information related to the input information into the second inference model and executes the second recognition processing.

5. The sensing information recognition system described in claim 1, characterized in that the first processing device uses the result of the first recognition processing to perform a predetermined process required to control an actuator that moves the vehicle, between the time when the first processing device executes the first recognition processing and the time when the result of the second recognition processing executed by the second processing device is obtained.

6. The sensing information recognition system described in claim 5, characterized in that the first processing device determines the reliability of the result of the first recognition processing by determining whether the result of the first recognition processing is consistent with the result of the second recognition processing, and if the result of the first recognition processing is consistent with the result of the second recognition processing, transmits the result of the specified processing to a control device that controls the actuator, and if the result of the first recognition processing is not consistent with the result of the second recognition processing, discards the result of the specified processing.

7. The sensing information recognition system according to claim 5, wherein the sensors are a pair of image sensors provided in a stereo camera device, the sensing information is stereo images acquired from the pair of image sensors, the first processing device generates a parallax image from the stereo images and detects an area where an arbitrary object is present in the parallax image as the first recognition processing, the second processing device identifies the type of the arbitrary object as the second recognition processing, the first processing device determines whether the arbitrary object present in the area where the arbitrary object is present detected by the first recognition processing is a reliable object that should be used to control the actuator based on the identification result of the arbitrary object that is the result of the second recognition processing, if the arbitrary object present in the area where the arbitrary object is present detected by the first recognition processing is a reliable object, transmits the result of the predetermined processing to a control device that controls the actuator, and if the arbitrary object present in the area where the arbitrary object is present detected by the first recognition processing is not a reliable object, discards the result of the predetermined processing.

Images