Sensor noise removal device and sensor noise removal method

Abstract

To provide a sensor noise removal device that can replace sensor data whose reliability has been reduced by noise with sensor data in which no noise is generated.SOLUTION: A sensor noise removal device comprises: a sensor data acquisition unit (11) that acquires sensor data related to a situation around a vehicle; a noise determination unit (12) that determines whether noise is generated in the sensor data acquired by the sensor data acquisition unit (11); and a data replacement unit (13) that, for the sensor data in which generation of noise is determined by the noise determination unit (12), estimates sensor data in which no noise is generated to generate data for replacement corresponding to a noise portion, and replaces the noise portion with the generated data for replacement.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a sensor noise removal device and a sensor noise removal method. [Background technology]

[0002] In processing based on sensor data acquired from a sensor, it is desirable that the acquired sensor data be reliable in order for the processing to be performed appropriately. For example, if the acquired sensor data contains noise, the sensor data becomes unreliable and may not be processed appropriately. BACKGROUND ART Conventionally, when performing processing based on sensor data acquired from a sensor, a technique is known in which sensor data with less noise is used from among the acquired sensor data (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2020-91281 Summary of the Invention [Problem to be solved by the invention]

[0004] On the other hand, some processes based on sensor data require the acquired sensor data without any missing parts when the process is performed. For example, when processing is performed using images acquired from a camera, the acquired image is required without any missing parts. In this case, there is a problem that even if the acquired sensor data is unreliable due to noise, there is no choice but to use the sensor data as is. It should be noted that the above-mentioned conventional technology is a technology that does not use sensor data in which noise is generated, and therefore the conventional technology cannot solve the above-mentioned problem.

[0005] The present disclosure has been made to solve the above-mentioned problems, and aims to provide a sensor noise removal device that can convert sensor data whose reliability has been reduced due to noise into sensor data in a noise-free state. [Means for solving the problem]

[0006] The sensor noise removal device according to the present disclosure includes a sensor data acquisition unit that acquires a plurality of sensor data relating to a surrounding situation of a vehicle, each of the sensor data being acquired by a plurality of sensors of different types, a noise determination unit that determines whether noise is occurring in the plurality of sensor data, and a first sensor data that is determined by the noise determination unit to be generating noise. , th 1 sensor data is acquired Tase A different species from the No Se Acquired by Reno By the size determination section Rino When there is second sensor data in which it is determined that noise does not occur, if an object existing in real space corresponding to the noise part of the first sensor data is not detected in the second sensor data, The first sensor data in which noise occurs is treated as the first sensor data in which noise does not occur. and a data substitution unit. [Effects of the Invention]

[0007] According to the present disclosure, sensor data whose reliability has been reduced due to noise can be converted into sensor data in a state where no noise is present. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an example of the configuration of a sensor noise removal device according to a first embodiment. [Figure 2]2A and 2B are diagrams for explaining an example of a replacement performed by a data replacement unit based on first distance data or second distance data in embodiment 1, where FIG. 2A shows an example of a captured image determined to contain noise before the data replacement unit performs replacement based on the first distance data or second distance data, and FIG. 2B shows an example of a replaced captured image after the data replacement unit performs replacement based on the first distance data or second distance data. [Figure 3] 3A is a diagram illustrating an example of an image of a captured image determined to contain noise before the data replacement unit performs replacement based on the first distance data or the second distance data in embodiment 1; FIG. 3B is a diagram illustrating an example of an image of a captured image after replacement as replaced sensor data after the data replacement unit performs replacement based on the first distance data or the second distance data. [Figure 4] 4 is a flowchart illustrating the operation of the sensor noise removal device according to the first embodiment. [Figure 5] 5 is a flowchart for explaining in detail the operation of the data replacing unit in step ST403 of FIG. 4. [Figure 6] 6A and 6B are diagrams illustrating an example of a hardware configuration of the sensor noise removal device according to the first embodiment. [Figure 7] FIG. 10 is a diagram illustrating an example of the configuration of a sensor noise removal device according to a second embodiment. [Figure 8] 10 is a flowchart illustrating the operation of the sensor noise removal device according to the second embodiment. [Figure 9] FIG. 10 is a diagram illustrating an example of the configuration of a sensor noise removal device according to a third embodiment. [Figure 10] FIG. 11 is a diagram illustrating an example of the configuration of a learning device according to a third embodiment. [Figure 11] FIG. 1 is a diagram illustrating an example of a neural network. [Figure 12]10 is a flowchart illustrating the operation of the sensor noise removal device according to the third embodiment. [Figure 13] 11 is a flowchart illustrating the operation of the learning device according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Embodiment 1 FIG. 1 is a diagram illustrating an example of the configuration of a sensor noise removal device 1 according to the first embodiment. In the first embodiment, it is assumed that the sensor noise elimination device 1 is mounted on a vehicle. The sensor noise elimination device 1 is connected to a plurality of types of sensors mounted on the vehicle, and acquires a plurality of sensor data relating to the surrounding conditions of the vehicle, each of which is acquired by the plurality of types of sensors. The sensor data relating to the surrounding conditions of the vehicle acquired by the sensors is used for various processes relating to the vehicle.

[0010] In some processes that use sensor data, the sensor data used cannot be substituted with other sensor data. In such cases, even if the sensor data used in the process contains noise and the other sensor data is normal sensor data without noise, the process will not be performed properly if the other sensor data is used. Conventionally, when a process is performed that uses sensor data that cannot be replaced by other sensor data, even if noise occurs in the sensor data to be used, the process has no choice but to use the sensor data that contains noise. For example, when displaying an image acquired by a camera capturing an image behind a vehicle or a camera mounted on a drive recorder on a display mounted on the vehicle, even if noise occurs in the acquired image, the only option was to display the image as is. Furthermore, for example, when processing is performed using artificial intelligence with certain sensor data as input, even if noise occurs in the input sensor data, there is no choice but to use the sensor data as input as is.

[0011] Therefore, when there is sensor data containing noise among the acquired multiple pieces of sensor data, the sensor noise removal device 1 according to the first embodiment converts the sensor data into sensor data in a state where no noise is present. Specifically, the sensor noise removal device 1 estimates sensor data in which no noise is present, generates data (hereinafter referred to as "replacement data") corresponding to the part in which noise is present (hereinafter referred to as "noise part"), and replaces the noise part of the sensor data in which noise is present with the generated replacement data. In the following first embodiment, replacing the noise part of the sensor data in which noise is present with sensor data in a state in which no noise is present is also simply referred to as "replacement."

[0012] In the first embodiment, the sensor data after the sensor noise removal device 1 has performed replacement to make the sensor data noise-free is referred to as “replaced sensor data.” Note that the sensor noise removal device 1 replaces the noise portion with replacement data during replacement, but this replacement does not change the characteristics of the data before replacement.

[0013] The sensor noise removal device 1 is only required to be configured to perform substitution when noise occurs in sensor data that cannot be substituted with other sensor data when processing using the sensor noise removal device 1 is performed.

[0014] 1, the multiple sensors are assumed to be a camera 21, a lidar 22, and a radar 23. Although the first embodiment assumes that three sensors are connected to the sensor noise elimination device 1, this is merely an example. The number of sensors connected to the sensor noise elimination device 1 may be two, four or more, or just one. The camera 21 captures an image of the periphery of the vehicle and outputs the captured image of the periphery of the vehicle (hereinafter referred to as a “captured image”) to the sensor noise removal device 1. The lidar 22 outputs point cloud data obtained by irradiating the periphery of the vehicle with laser light as distance data (hereinafter referred to as "first distance data") to the sensor noise removal device 1. The point cloud data indicates a distance vector and reflection intensity for each point where the laser light is reflected. The radar 23 scans and emits millimeter waves around the vehicle, and outputs distance data (hereinafter referred to as "second distance data") obtained based on the received radio waves to the sensor noise removal device 1. The second distance data indicates a distance vector for each point where the millimeter waves are reflected. The ranges in which the camera 21, the lidar 22, and the radar 23 detect the surrounding conditions of the vehicle overlap with each other. For example, the camera 21 captures an image behind the vehicle. The lidar 22 and the radar 23 detect objects present behind the vehicle.

[0015] In the first embodiment, when processing is performed using the captured image acquired from camera 21, the captured image cannot be substituted for the first distance data acquired from lidar 22 or the second distance data acquired from radar 23. It is also assumed that an event that causes noise may occur in camera 21. If an event that causes noise occurs in the camera 21, noise will occur in the captured image. An event that causes noise is, for example, water droplets, dirt, or insects adhering to the lens of the camera 21. In this case, blurring occurs in the captured image as noise. When noise occurs in the captured image, the sensor noise removal device 1 estimates a captured image in which the noise does not occur, generates replacement data corresponding to the pixels in the noisy portion, and replaces the noise portion of the captured image containing noise with the generated replacement data.

[0016] In the first embodiment, it is assumed that no event that causes noise occurs in the lidar 22 and the radar 23. In other words, it is assumed that no noise occurs in the first distance data and the second distance data. The replacement performed by the sensor noise removal device 1 will be described in detail later.

[0017] 1, the sensor noise removal device 1 according to the first embodiment includes a sensor data acquisition unit 11, a noise determination unit 12, a data replacement unit 13, an output unit 14, a sensor DB (database) 15, and a noise DB 16. The data replacement unit 13 includes a replacement possibility determination unit 131.

[0018] The sensor data acquisition unit 11 acquires sensor data relating to the surrounding conditions of the vehicle. Specifically, the sensor data acquisition unit 11 acquires an image captured by the camera 21, first distance data acquired by the lidar 22, and second distance data acquired by the radar 23. The sensor data acquisition unit 11 outputs the acquired captured image, first distance data, and second distance data to the noise determination unit 12. Furthermore, the sensor data acquiring unit 11 stores the acquired captured image, first distance data, and second distance data in the sensor DB 15. At this time, the sensor data acquiring unit 11 stores, for example, the captured image, the first distance data, and the second distance data in the sensor DB 15 in association with information on the data acquisition date and time.

[0019] The noise determination unit 12 determines whether or not noise occurs in the sensor data acquired by the sensor data acquisition unit 11. Specifically, in the first embodiment, the noise determining unit 12 determines whether or not noise occurs in the captured image acquired by the sensor data acquiring unit 11. For example, the noise determination unit 12 determines whether or not blurring occurs in the captured image using a known image recognition process. If blurring occurs in the captured image, the noise determination unit 12 determines that noise occurs in the captured image. Note that, for example, if blurring occurs in even one pixel in the captured image, the noise determination unit 12 determines that noise occurs in the captured image. If blurring does not occur in the captured image, the noise determination unit 12 determines that noise does not occur in the captured image.

[0020] The noise determination unit 12 outputs the captured image acquired from the sensor data acquisition unit 11, together with the determination result as to whether or not the captured image contains noise, to the data replacement unit 13. At this time, the noise determination unit 12 also outputs the first distance data and the second distance data acquired from the sensor data acquisition unit 11 to the data replacement unit 13.

[0021] The data replacing unit 13 estimates noise-free sensor data for sensor data determined by the noise determining unit 12 to contain noise, generates replacement data corresponding to the noise portion of the sensor data, and replaces the noise portion with the generated replacement data. In the first embodiment, the data replacing unit 13 estimates noise-free captured images for captured images determined by the noise determining unit 12 to contain noise, generates replacement data corresponding to the noise portion, and replaces the noise portion with the generated replacement data.

[0022] Specifically, first, the replacement possibility determination unit 131 of the data replacement unit 13 determines whether or not the conditions (hereinafter referred to as "replaceability conditions") that enable replacement of the noise portion in the sensor data that has been determined by the noise determination unit 12 to contain noise are satisfied, thereby determining whether or not replacement is possible for the captured image that has been determined to contain noise. When the replacement possibility determination unit 131 determines that replacement is possible, the data replacement unit 13 generates replacement data and replaces the noise part of the captured image that has been determined by the noise determination unit 12 to contain noise with the generated replacement data.

[0023] Here, the replaceable conditions include a first replaceable condition and a second replaceable condition. The first replaceable condition is set as a condition that allows replacement of the noise portion in the sensor data only from the sensor data determined by the noise determination unit 12 to contain noise. The first replaceable condition is, for example, that when the sensor data containing noise is a captured image, the number of pixels containing noise is equal to or less than a preset threshold value (hereinafter referred to as the "replaceability determination threshold value").

[0024] The second replaceable condition is set as a condition that enables replacement of the noise portion in sensor data that is determined by the noise determination unit 12 to contain noise based on sensor data that is determined by the noise determination unit 12 to contain no noise, among the multiple sensor data acquired by the sensor data acquisition unit 11. The second substitutable condition is, for example, that there is other sensor data without noise that has been acquired in a real space corresponding to the range in which noise occurs in the sensor data in which noise occurs.

[0025] The replacement possibility determining unit 131 first determines whether or not the first replacement possibility condition is satisfied. For example, if the first replaceable condition is as in the example described above, the replaceability determination unit 131 first determines whether the number of pixels in which noise has occurred in the captured image in which noise has been determined to have occurred by the noise determination unit 12 is equal to or less than the threshold for determining whether replacement is possible. If the number of pixels where noise occurs is equal to or less than the threshold for determining whether or not replacement is possible, the replacement possibility determination unit 131 determines that the first replacement condition is satisfied and that the noise portion in the captured image can be replaced only from the captured image determined by the noise determination unit 12 to contain noise. The replacement possibility determination unit 131 outputs to the data replacement unit 13 information indicating that replacement is possible only from the captured image determined by the noise determination unit 12 to contain noise. If the number of pixels where noise occurs is greater than the threshold value for determining whether or not replacement is possible, the first replaceable condition is not satisfied, and therefore the replacement possibility determination unit 131 determines that it is impossible to replace the noise part in the captured image based only on the captured image where noise is determined to occur. This is because if the part where noise occurs is large, it is difficult to predict what the captured image would look like if noise did not occur in the noisy part.

[0026] If it is determined that the first substitutable condition is not satisfied, the substitutability determining unit 131 determines whether or not the second substitutable condition is satisfied. For example, if the second replaceable condition is as in the example described above, the replaceability determination unit 131 determines whether or not there is first distance data or second distance data acquired for the real space in the range where noise occurs in the captured image. As described above, the ranges in which the camera 21, the LIDAR 22, and the radar 23 detect the vehicle's surrounding conditions overlap with each other. It is assumed that the installation positions of the camera 21, the LIDAR 22, and the radar 23, and the ranges in which the camera 21, the LIDAR 22, and the radar 23 can detect the vehicle's surrounding conditions, are known in advance. In this case, the replacement possibility determination unit 131 can identify the first distance data or the second distance data corresponding to the range in which noise occurs in the captured image.

[0027] If there is first distance data or second distance data corresponding to the range where noise occurs in the captured image, the replacement possibility determination unit 131 determines that the second replacement condition is satisfied and that replacement is possible based on the sensor data, in other words, the first distance data or the second distance data, among the multiple sensor data acquired by the sensor data acquisition unit 11, for which the noise determination unit 12 has determined that no noise occurs. The replacement possibility determination unit 131 outputs to the data replacement unit 13 information indicating that replacement is possible based on the sensor data that the noise determination unit 12 has determined not to contain noise, in other words, the first distance data or the second distance data, among the multiple sensor data acquired by the sensor data acquisition unit 11.

[0028] If the replacement possibility determination unit 131 determines that neither the first replaceable condition nor the second replaceable condition is satisfied, it determines that replacement is not possible for the captured image determined to contain noise by the noise determination unit 12. The replacement possibility determination unit 131 outputs information to the effect that replacement is not possible to the data replacement unit 13.

[0029] When the replacement possibility determination unit 131 outputs information indicating that replacement is possible only from captured images determined by the noise determination unit 12 to contain noise, the data replacement unit 13 generates replacement data by inferring a noise-free captured image based on the captured image determined to contain noise. Then, the data replacement unit 13 replaces the noise portion of the captured image with the generated replacement data. Specifically, for example, for a pixel included in a noise portion, the data replacement unit 13 generates replacement data from pixels (hereinafter referred to as "neighboring pixels") that are adjacent to the pixel and do not contain noise, and replaces the pixel in the noise portion with the generated replacement data. More specifically, the data replacement unit 13 may, for example, estimate that in a noise-free captured image, the noise portion will have pixel values ​​close to those of adjacent pixels, and generate replacement data whose pixel value is the average of the pixel values ​​of the adjacent pixels. The range of pixels that constitutes adjacent pixels is predetermined. For example, the data replacement unit 13 may calculate the difference between each adjacent pixel and the average pixel value of the noise portion, extract adjacent pixels whose difference is less than a predetermined threshold, and generate replacement data whose pixel value is the average of the pixel values ​​of the extracted adjacent pixels. This allows the data replacement unit 13 to generate replacement data based on adjacent pixels that are estimated to be more closely related to the pixel values ​​of the noise portion. For example, the data replacement unit 13 may, for example, estimate that in a noise-free captured image, a pixel value identical to the pixel value of an adjacent pixel in the noise portion will be consecutive, and generate replacement data whose pixel value is the same as the adjacent pixel value. For example, if the noise portion is a narrow range, such as one pixel, the data replacement unit 13 may generate replacement data by removing noise from the pixels in the noise portion using a known super-resolution technique. In this way, the data replacement unit 13 generates replacement data based on adjacent pixels or pixels in the noise part, and replaces the pixels in the noise part with the replacement data, thereby generating a captured image as replaced sensor data (hereinafter referred to as the "replaced captured image") in which the noise part is replaced with an image that is presumed to have been captured in a noise-free state.

[0030] On the other hand, when information indicating that replacement is possible based on the first distance data or the second distance data determined by the noise determination unit 12 as not having noise is output from the replacement possibility determination unit 131, the data replacement unit 13 generates replacement data by inferring a noise-free captured image based on the first distance data or the second distance data from among the plurality of sensor data acquired by the sensor data acquisition unit 11. Then, the data replacement unit 13 replaces the noise portion of the captured image determined by the noise determination unit 12 as having noise with the generated replacement data.

[0031] The following describes an example of the replacement performed by the data replacement unit 13 based on the first distance data or the second distance data. FIG. 2 is a diagram illustrating an example of replacement performed by data replacement unit 13 based on the first distance data or the second distance data in the first embodiment. Figure 2A is a diagram showing an example of an image of a captured image that has been determined to contain noise before the data replacement unit 13 performs replacement based on the first distance data or the second distance data, and Figure 2B is a diagram showing an example of an image of a captured image after the data replacement unit 13 performs replacement based on the first distance data or the second distance data. In the captured image shown in FIG. 2A, the range indicated by 201 to 203 is the range where blurring occurs due to noise.

[0032] The data substitution unit 13 first estimates whether an object is detected in the noise portion of the captured image, in other words, in each of the ranges 201 to 203 in Fig. 2A, based on the first distance data or the second distance data. For example, if an object existing in real space corresponding to the noise portion of the captured image is detected in the first distance data or the second distance data, the data substitution unit 13 estimates that an object is also detected in the captured image. If an object existing in real space corresponding to the noise portion of the captured image is not detected in the first distance data or the second distance data, the data substitution unit 13 estimates that an object is not detected in the captured image. As an example, if no object is detected in the first distance data and the second distance data, the data replacement unit 13 infers that no object is detected in the noise portion of the captured image. In this case, for example, for a pixel included in a noise portion, the data replacement unit 13 generates replacement data from neighboring pixels that are not generating noise and are adjacent to the pixel, and replaces the pixel in the noise portion with the generated replacement data. The details of generating replacement data from neighboring pixels that are not generating noise and replacing the pixel in the noise portion with the generated replacement data have already been explained, so a duplicate explanation will be omitted.

[0033] As a result, the data replacement unit 13 generates a replaced captured image in which the noise-generated range indicated by 201 to 203 in Fig. 2A is replaced with a blur-free image, as shown in Fig. 2B, for example. In Fig. 2B, the pixels in the portion indicated by 201 to 203 in Fig. 2A are replaced with blur-free pixels that are assumed to be from an image captured in a state where no object is present. For convenience, in FIG. 2B, the outer frames of the noise portions indicated by 201 to 203 in FIG. 2A are shown by dotted lines.

[0034] In the above example, the data replacement unit 13 infers that no object is detected in the noise portion of the captured image, but this is just one example. An example of replacement by the data replacement unit 13 when the data replacement unit 13 estimates that an object is detected in a noise portion of a captured image will be described below. FIG. 3 is a diagram illustrating an image of another example of replacement performed by data replacement section 13 based on the first distance data or the second distance data in the first embodiment. FIG. 3A is a diagram showing an example of an image of a captured image determined to contain noise before the data replacement unit 13 performs replacement based on the first distance data or the second distance data, and FIG. 3B is a diagram showing an example of an image of a captured image after replacement as replaced sensor data after the data replacement unit 13 performs replacement based on the first distance data or the second distance data.

[0035] For example, if an object existing in real space corresponding to a noise portion of the captured image is detected in the first distance data or the second distance data, the data replacement unit 13 infers that the object is also detected in the captured image. In this case, the data replacement unit 13 generates replacement data so as to indicate the object that is inferred to have been detected. Here, for example, it is assumed that a person is detected in the real space corresponding to the noise portion indicated by 301 in Fig. 3A in the first distance data or the second distance data. Also, for example, it is assumed that a car is detected in the real space corresponding to the noise portion indicated by 302 in Fig. 3A in the first distance data or the second distance data. In this case, the data replacement unit 13 infers that a person is detected in the noise portion indicated by 301 in Fig. 3A and a car is detected in the noise portion indicated by 302 in Fig. 3A in the captured image, and generates replacement data so that the person is shown in the noise portion indicated by 301 in Fig. 3A and the car is shown in the noise portion indicated by 302 in Fig. 3A. In this case, data replacement unit 13 does not need to generate replacement data that accurately reproduces the object detected in the first distance data or the second distance data. Data replacement unit 13 only needs to generate replacement data that indicates the position, type, or orientation of the detected object. Data replacement unit 13 does not need to generate replacement data that indicates, for example, the color of the detected object.

[0036] As a result, the data replacement unit 13 replaces, for example, 301 to 303 in FIG. 3A as shown in FIG. 3B. 3 A replaced captured image is generated in which the area where noise occurs, shown as 03, is de-blurred. In Fig. 3B, no blurring occurs in the noise area indicated by 301 in Fig. 3A, and a person is drawn (see 304 in Fig. 3B). Also, in Fig. 3B, no blurring occurs in the noise area indicated by 302 in Fig. 3A, and a car is drawn (see 305 in Fig. 3B). In addition, the noise portion indicated by 303 in FIG. 3A is replaced with non-blurred pixels that are assumed to be from an image captured when there is no object, because the data replacement unit 13 has assumed that no object has been detected. In FIG. 3B, for the sake of convenience, the outer frames of the noise portions indicated by 301 to 303 in FIG. 3A are shown by dotted lines.

[0037] As explained using Figures 2 and 3, the data replacement unit 13 generates replacement data based on the first distance data or the second distance data, and replaces pixels in the noise part with the replacement data, thereby generating a replaced captured image in which the noise part is replaced with an image that is presumed to have been captured in a noise-free state.

[0038] In addition, when the replacement possibility determination unit 131 outputs information indicating that replacement is not possible, the data replacement unit 13 stores in the noise DB 16, as replacement impossible information, information that associates the captured image determined to contain noise, information indicating that the captured image is not replaceable, and information that can identify the noise portion in the captured image where noise is occurring. By storing the unreplaceable information, the next time the replaceability determining unit 131 refers to the unreplaceable information, it can determine whether or not the captured image can be replaced.

[0039] When the data substitution unit 13 has performed substitution on the captured image, it outputs the substituted captured image to the output unit 14. When the data substitution unit 13 has not performed substitution on the captured image, it outputs the captured image acquired by the sensor data acquisition unit 11 to the output unit 14. In addition, the data substitution unit 13 outputs the first distance data and the second distance data acquired by the sensor data acquisition unit 11 to the output unit 14.

[0040] The output unit 14 outputs the sensor data output from the data substitution unit 13. Specifically, the output unit 14 outputs the substituted captured image or the captured image output from the data substitution unit 13, the first distance data, and the second distance data. The output destination of each sensor data is a device that performs processing using the sensor data. For example, when a display (not shown) mounted on a vehicle displays the captured image, the output unit 14 outputs the replaced captured image or the captured image to the display.

[0041] The sensor DB 15 stores the sensor data acquired by the sensor data acquisition unit 11. 1, the sensor DB 15 is provided in the sensor noise removal device 1, but this is merely an example. The sensor DB 15 may be provided outside the sensor noise removal device 1 in a location that can be referenced by the sensor noise removal device 1.

[0042] The noise DB 16 stores non-replaceable information. The noise DB 16 may store, as initial data, captured images generated when a driving simulation is performed for each vehicle model, or captured images obtained by the camera 21 during a test drive. When the noise DB 16 stores the above-described initial data, the data replacement unit 13 may generate replacement data based on the captured image stored in the noise DB 16 when performing replacement. For example, when the data replacement unit 13 estimates that no object has been detected in a noise portion of the captured image determined by the noise determination unit 12 to contain noise, the data replacement unit 13 extracts initial data from a range corresponding to the noise portion and generates replacement data. Also, when the data replacement unit 13 estimates that an object has been detected in a noise portion of the captured image determined by the noise determination unit 12 to contain noise, the data replacement unit 13 extracts initial data from a range corresponding to the noise portion and superimposes the object estimated to have been detected on the initial data to generate replacement data. 1, the noise DB 16 is assumed to be provided in the sensor noise removal device 1, but this is merely an example. The noise DB 16 may also be provided in a location outside the sensor noise removal device 1 that can be referenced by the sensor noise removal device 1.

[0043] The operation of the sensor noise removal device 1 according to the first embodiment will be described. FIG. 4 is a flowchart for explaining the operation of the sensor noise removal device 1 according to the first embodiment.

[0044] The sensor data acquisition unit 11 acquires sensor data relating to the surrounding conditions of the vehicle (step ST401). Specifically, the sensor data acquisition unit 11 acquires an image captured by the camera 21, first distance data acquired by the lidar 22, and second distance data acquired by the radar 23. The sensor data acquisition unit 11 outputs the acquired captured image, first distance data, and second distance data to the noise determination unit 12. The sensor data acquiring unit 11 also stores the acquired captured image, first distance data, and second distance data in the sensor DB 15.

[0045] The noise determination unit 12 determines whether or not noise occurs in the sensor data acquired by the sensor data acquisition unit 11 in step ST401 (step ST402). Specifically, the noise determination unit 12 determines whether or not noise occurs in the captured image acquired by the sensor data acquisition unit 11. The noise determination unit 12 outputs the captured image acquired from the sensor data acquisition unit 11, together with the determination result as to whether or not the captured image contains noise, to the data replacement unit 13. At this time, the noise determination unit 12 also outputs the first distance data and the second distance data acquired from the sensor data acquisition unit 11 to the data replacement unit 13.

[0046] The data replacing unit 13 estimates noise-free sensor data for the sensor data determined by the noise determining unit 12 in step ST402 to contain noise, generates replacement data corresponding to the noise portion of the sensor data, and replaces the noise portion with the generated replacement data (step ST403). Specifically, the data replacing unit 13 estimates noise-free captured images for the captured images determined by the noise determining unit 12 to contain noise, generates replacement data corresponding to the noise portion, and replaces the noise portion with the generated replacement data. When the data substitution unit 13 has performed substitution on the captured image, it outputs the substituted captured image to the output unit 14. When the data substitution unit 13 has not performed substitution on the captured image, it outputs the captured image acquired by the sensor data acquisition unit 11 to the output unit 14. In addition, the data substitution unit 13 outputs the first distance data and the second distance data acquired by the sensor data acquisition unit 11 to the output unit 14.

[0047] The output unit 14 outputs the sensor data output from the data substitution unit 13 in step ST403 (step ST404). Specifically, the output unit 14 outputs the substituted captured image or the captured image output from the data substitution unit 13, the first distance data, and the second distance data.

[0048] FIG. 5 is a flowchart for explaining in detail the operation of the data substitution section 13 in step ST403 of FIG.

[0049] The replacement possibility determination unit 131 determines whether or not the captured image in which noise has been determined to have occurred by the noise determination unit 12 in step ST402 of Figure 4 satisfies the first replacement condition, and thereby determines whether or not the noise portion in the captured image can be replaced using only the captured image in which noise has been determined to have occurred (step ST501).

[0050] In step ST501, if the replacement possibility determination unit 131 determines that the first replacement condition is satisfied, that is, if it determines that the noise portion in the captured image can be replaced only from the captured image that has been determined to contain noise (if "YES" in step ST501), it outputs information to the data replacement unit 13 that replacement is possible only from the captured image that has been determined to contain noise by the noise determination unit 12.

[0051] The data replacing unit 13 generates replacement data by estimating a noise-free captured image based on the captured image determined to contain noise, and then replaces the noise part of the captured image with the generated replacement data (step ST502).

[0052] On the other hand, if it is determined in step ST501 that the first replaceable condition is not satisfied, that is, if it is determined that it is not possible to replace the noise portion in the captured image only from the captured image in which noise is determined to have occurred (if "NO" in step ST501), the replacement possibility determination unit 131 performs the operation of step ST503. In step ST503, the replacement possibility determination unit 131 determines whether the second replacement condition is satisfied, thereby determining whether it is possible to replace the noise part of the captured image based on the first distance data or the second distance data among the multiple sensor data acquired by the sensor data acquisition unit 11 in step ST401 of Figure 4 (step ST503).

[0053] In step ST503, if the replacement possibility determination unit 131 determines that the second replacement condition is satisfied, that is, if it determines that it is possible to replace the noise part of the captured image based on the first distance data or the second distance data (if "YES" in step ST503), it outputs information to the data replacement unit 13 that replacement is possible based on the first distance data or the second distance data.

[0054] The data replacing unit 13 generates replacement data by estimating a noise-free captured image based on the first distance data or the second distance data in which the noise determining unit 12 has determined that noise has not occurred, among the plurality of sensor data acquired by the sensor data acquiring unit 11 in step ST401 in Fig. 4. Then, the data replacing unit 13 replaces the noise part of the captured image in which the noise determining unit 12 has determined that noise has occurred, with the generated replacement data (step ST504).

[0055] If it is determined in step ST503 that the second replaceable condition is not satisfied, that is, if it is determined that it is impossible to replace the noise part of the captured image based on the first distance data or the second distance data (if "NO" in step ST503), the replaceability determination unit 131 outputs information to the effect that replacement is impossible to the data replacement unit 13.

[0056] The data replacement unit 13 stores the unreplaceable information in the noise DB 16 (step ST505).

[0057] In this way, when the sensor noise removal device 1 according to the first embodiment determines that noise is present in sensor data (captured image) relating to the surrounding conditions of the vehicle, it estimates noise-free sensor data for the sensor data determined to contain noise, generates replacement data corresponding to the noise portion, and replaces the noise portion with the generated replacement data. In this way, the sensor noise removal device 1 can convert sensor data whose reliability has been reduced due to noise into sensor data in a noise-free state.

[0058] In the first embodiment described above, the data replacement unit 13 has a function of generating replacement data based on a captured image determined to contain noise and replacing the noise portion of the captured image with the generated replacement data (hereinafter referred to as the "first replacement function"), and a function of generating replacement data based on first distance data or second distance data determined to contain no noise and replacing the noise portion of the captured image with the replacement data (hereinafter referred to as the "second replacement function"), but this is merely an example. The data replacement unit 13 may be configured to have either the first replacement function or the second replacement function. When the data substitution unit 13 has only the first substitution function, the substitution possibility determination unit 131 only determines whether or not the first substitution possibility condition is satisfied. In this case, the operations of steps ST503 and ST504 in the operation of the sensor noise removal device 1 described with reference to FIG. 5 are omitted. Furthermore, when the data replacement unit 13 has only the second replacement function, the replacement possibility determination unit 131 only determines whether or not the second replacement possibility condition is satisfied. In this case, the operations of steps ST501 and ST502 in the operation of the sensor noise removal device 1 described with reference to FIG. 5 are omitted.

[0059] In the first embodiment, the data replacement unit 13 is provided with the replacement possibility determination unit 131, but the replacement possibility determination unit 131 is not essential. For example, the data replacement unit 13 may have the function of the replacement possibility determination unit 131, and the data replacement unit 13 may determine whether or not the replacement possibility condition is satisfied when performing replacement.

[0060] In the first embodiment described above, it is assumed that noise may occur in the captured image, but this is merely an example. In the first embodiment, it may also be assumed that noise may occur in the first distance data and the second distance data. The noise determination unit 12 can determine whether or not noise is occurring in all of the sensor data acquired by the sensor data acquisition unit 11. For example, the noise determination unit 12 can determine whether noise has occurred in the first distance data or the second distance data. Specifically, for example, if any one of the first distance data, more specifically, the point cloud data included in the first distance data, indicates "0", the noise determination unit 12 determines that noise has occurred in the first distance data. Also, if the second distance data indicates "0", the noise determination unit 12 determines that noise has occurred in the second distance data. The first replaceable condition when the sensor data is not an image is, for example, that when the sensor data is first distance data, data indicating "0" in point cloud data obtained by irradiating the periphery of the vehicle with laser light is equal to or less than a preset threshold. In this case, for example, it is assumed that the first replaceable condition is satisfied and the replaceability determination unit 131 outputs information indicating that replacement is possible only from the first distance data determined by the noise determination unit 12 to contain noise. In this case, the data replacement unit 13 generates replacement data from noise-free data for data included in noise portions of the point cloud data, and replaces the data in the noise portions with the generated replacement data.

[0061] In the first embodiment, the noise determining unit 12 may determine whether or not noise occurs in the sensor data based on the characteristics of the sensor data. The sensor data may have characteristics that are affected by the environment, etc. When the sensor data is affected by the environment, etc., the sensor data may not show normal values. For example, if the sensor data is a captured image, the captured image has a characteristic of being affected by the high beams of an oncoming vehicle, the light of a street lamp, etc. When there is the high beams of an oncoming vehicle or the light of a street lamp, so-called overexposure occurs in the captured image in a portion that is exposed to the high beams or the light of the street lamp, etc. If there is a pixel in the captured image whose brightness is equal to or greater than a preset threshold, the noise determination unit 12 determines that the pixel is affected by the light of the high beams or the street lamp, etc., and determines that the overexposed portion is a noise portion affected by the high beams, etc. Furthermore, for example, captured images have the characteristic of being affected by weather or time of day. For example, in bad weather such as fog, or at night, the captured image may be unclear. If there are pixels in the captured image whose clarity is equal to or less than a preset threshold, the noise determination unit 12 determines that the captured image has been affected by weather or time of day, and determines the pixel portion whose clarity is equal to or less than the threshold as a noise portion. The noise determination unit 12 may obtain information about the weather, for example, from a weather database (not shown) that stores information about the weather, or from a website. The noise determination unit 12 may obtain information about time of day, for example, from a clock (not shown) installed in the vehicle.

[0062] Furthermore, for example, the first distance data and the second distance data have a characteristic that they are affected by water. When the sensor data is the first distance data or the second distance data, for example, if there is a waterfall around the vehicle, the laser light emitted from the lidar 22 or the millimeter waves emitted from the radar 23 will pass through the waterfall, and the first distance data and the second distance data will not be acquired correctly. For example, if there is a waterfall around the vehicle, the noise determination unit 12 determines that the first distance data and the second distance data have been affected by the waterfall and that noise has occurred in the first distance data and the second distance data. Note that the noise determination unit 12 may acquire information that there is a waterfall around the vehicle from, for example, a map information DB (not shown).

[0063] Information regarding what kind of environment, etc. will affect each piece of sensor data (hereinafter referred to as "characteristic definition information") is set in advance and stored in a location that can be referenced by the noise determination unit 12. The noise determination unit 12 references the characteristic definition information to determine the environment, etc. that should be taken into consideration for the sensor data. Then, the noise determination unit 12 takes the environment, etc. into consideration and determines whether noise is occurring in the sensor data.

[0064] In this way, the sensor noise removal device 1 can also be configured to determine whether noise is occurring in the sensor data based on the characteristics of the sensor data acquired by the sensor data acquisition unit 11. This allows the sensor noise removal device 1 to determine whether noise is occurring in the sensor data, taking into account the characteristics of the sensor data.

[0065] 6A and 6B are diagrams illustrating an example of a hardware configuration of the sensor noise removing device 1 according to the first embodiment. In the first embodiment, the functions of the sensor data acquisition unit 11, the noise determination unit 12, the data replacement unit 13, and the output unit 14 are realized by the processing circuit 601. That is, the sensor noise removal device 1 includes the processing circuit 601 for controlling, when noise occurs in the acquired sensor data, to estimate noise-free sensor data for the noisy sensor data, generate replacement data corresponding to the noise portion, and replace the noise portion with the generated replacement data, when noise occurs in the acquired sensor data. The processing circuit 601 may be dedicated hardware as shown in FIG. 6A, or may be a CPU (Central Processing Unit) 604 that executes a program stored in a memory 605 as shown in FIG. 6B.

[0066] When the processing circuitry 601 is dedicated hardware, the processing circuitry 601 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

[0067] When the processing circuit 601 is a CPU 604, the functions of the sensor data acquisition unit 11, the noise determination unit 12, the data substitution unit 13, and the output unit 14 are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in the memory 605. The processing circuit 601 executes the functions of the sensor data acquisition unit 11, the noise determination unit 12, the data substitution unit 13, and the output unit 14 by reading and executing the program stored in the memory 605. That is, the sensor noise removal device 1 includes the memory 605 for storing a program that, when executed by the processing circuit 601, results in the execution of steps ST401 to ST404 in FIG. 4 described above. It can also be said that the program stored in the memory 605 causes a computer to execute the procedures or methods of the sensor data acquisition unit 11, the noise determination unit 12, the data substitution unit 13, and the output unit 14. Here, memory 605 refers to, for example, non-volatile or volatile semiconductor memory such as RAM, ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), or magnetic disk, flexible disk, optical disk, compact disk, mini disk, DVD (Digital Versatile Disc), etc.

[0068] It is also possible to realize some of the functions of the sensor data acquisition unit 11, the noise determination unit 12, the data substitution unit 13, and the output unit 14 with dedicated hardware and some with software or firmware. For example, the functions of the sensor data acquisition unit 11 and the output unit 14 can be realized by a processing circuit 601 as dedicated hardware, and the functions of the noise determination unit 12 and the data substitution unit 13 can be realized by the processing circuit 601 reading and executing a program stored in a memory 605. The sensor DB 15 and the noise DB 16 use the memory 605. This is just an example, and the sensor DB 15 and the noise DB 16 may be configured with an HDD, an SSD (Solid State Drive), a DVD, or the like. The sensor noise removal device 1 also includes an input interface device 602 and an output interface device 603 that perform wired or wireless communication with devices such as the camera 21, the lidar 22, or the radar 23.

[0069] As described above, according to the first embodiment, the sensor noise removal device 1 is configured to include the sensor data acquisition unit 11 that acquires sensor data related to the surrounding conditions of the vehicle, the noise determination unit 12 that determines whether noise is present in the sensor data acquired by the sensor data acquisition unit 11, and the data replacement unit 13 that, for sensor data determined by the noise determination unit 12 to contain noise, estimates noise-free sensor data, generates replacement data corresponding to the noise portion, and replaces the noise portion with the generated replacement data. Therefore, the sensor noise removal device 1 can convert sensor data whose reliability has been reduced due to noise into sensor data in a noise-free state.

[0070] Furthermore, the sensor noise removal device 1 includes a replacement possibility determination unit 131 that determines whether or not replacement of a noise portion is possible in sensor data determined by the noise determination unit 12 to contain noise is possible, and when the replacement possibility determination unit 131 determines that replacement is possible, the data replacement unit 13 replaces the noise portion of the sensor data determined by the noise determination unit 12 to contain noise with replacement data. When the replacement possibility determination unit 131 determines that replacement is not possible, it can next time refer to replacement impossibility information to that effect and determine whether or not replacement can be performed on the sensor data determined to contain noise.

[0071] Furthermore, in the sensor noise removal device 1, the sensor data acquisition unit 11 acquires a plurality of pieces of sensor data, and the data replacement unit 13 generates replacement data by inferring noise-free sensor data based on sensor data that the noise determination unit 12 has determined not to contain noise among the plurality of pieces of sensor data acquired by the sensor data acquisition unit 11, and replaces the noise portion of the sensor data that has been determined to contain noise by the noise determination unit 12 with the generated replacement data. Therefore, the sensor noise removal device 1 can convert sensor data whose reliability has been reduced due to noise into sensor data in a state where no noise is present.

[0072] Furthermore, in the sensor noise removal device 1, the data replacement unit 13 predicts whether an object is detected in the noise portion of the sensor data determined by the noise determination unit 12 to contain noise, based on the sensor data determined to contain no noise by the noise determination unit 12, and if it predicts that the object is detected, generates replacement data as data that indicates the position, type, or orientation of the object. Therefore, the sensor noise removal device 1 can convert sensor data whose reliability has been reduced due to noise into sensor data in a noise-free state so that the object appears when it is predicted that an object has been detected in the noise portion based on the sensor data determined to contain no noise.

[0073] Furthermore, in the sensor noise removal device 1, the data replacement unit 13 generates replacement data by inferring sensor data that is not affected by noise based on the sensor data that has been determined to have noise by the noise determination unit 12, and replaces the noise portion of the sensor data that has been determined to have noise by the noise determination unit 12 with the generated replacement data. Therefore, the sensor noise removal device 1 can convert sensor data whose reliability has been reduced due to noise into sensor data that is not affected by noise.

[0074] Furthermore, in the sensor noise removal device 1, the noise determination unit 12 determines whether or not noise has occurred in the sensor data based on the characteristics of the sensor data acquired by the sensor data acquisition unit 11. Therefore, the sensor noise removal device 1 can take into account the characteristics of the sensor data and convert sensor data whose reliability has been reduced due to noise into sensor data in a noise-free state.

[0075] Embodiment 2 In addition to the functions described in the first embodiment, the sensor noise removal device may have a function of detecting an object based on acquired sensor data and determining the validity of the object detected in multiple sensor data. In the second embodiment, an embodiment having a function of determining the validity of an object detected in a plurality of sensor data will be described.

[0076] FIG. 7 is a diagram illustrating an example of the configuration of a sensor noise removing device 1a according to the second embodiment. The sensor noise removal device 1a according to the second embodiment is mounted on a vehicle and connected to a camera 21, a lidar 22, and a radar 23, similar to the sensor noise removal device 1 according to the first embodiment. 7, the same components as those of the sensor noise removal device 1 described in the first embodiment with reference to FIG. 1 are denoted by the same reference numerals, and redundant description will be omitted. The sensor noise removal device 1a according to the second embodiment differs from the sensor noise removal device 1 according to the first embodiment in that it includes an object detection unit 17, a detection result determination unit 18, and a detection result correction unit 19.

[0077] Object detection unit 17 detects an object for each piece of sensor data acquired by sensor data acquisition unit 11. In the second embodiment, object detection unit 17 detects an object for each of the captured image, first distance data, and second distance data acquired by sensor data acquisition unit 11. The object detection unit 17 may detect an object using a known technique. Object detection unit 17 outputs information about the object detection result (hereinafter referred to as "object detection result information") for each piece of sensor data to detection result determination unit 18. The object detection result information includes at least the sensor data that detected the object, the position of the detected object, the type of the object, and information that can identify the orientation of the object.

[0078] The detection result determination unit 18 determines the validity of the object detection result by the object detection unit 17 based on the object detection result information output from the object detection unit 17 . For example, suppose that a car with a picture of a person drawn on it is currently present within the object detection ranges of the camera 21, the lidar 22, and the radar 23. Then, suppose that the object detection unit 17 detects the person based on the captured image, detects the car based on the first distance data, and detects the car based on the second distance data.

[0079] For example, based on the fact that a car is detected from the first distance data and the second distance data, while a person is detected from the captured image, the detection result determination unit 18 determines that the validity of the detection result that the first distance data and the second distance data detected a car is high, and that the validity of the detection result that the captured image detected a person is low. In this way, the detection result determination unit 18 compares the objects detected from the multiple sensor data, and if, for example, the object detected from one sensor data is different from the objects detected from the other multiple sensor data, it determines that the validity of the object detection result based on the one sensor data is low, even though the objects detected from the other multiple sensor data are the same. For example, if the objects detected from the multiple sensor data are all different, the detection result determination unit 18 determines that the object detection result is indeterminable. Taking the above example, suppose that the object detection unit 17 detects a person based on the captured image, a car based on the first distance data, and a signboard based on the second distance data. In this case, the detection result determination unit 18 determines that the object detection result is indeterminable. For example, the detection result determination unit 18 may determine that the validity of the detection result that detected a certain object is high if the ratio of the number of detected objects to the total number of objects detected from multiple sensor data is equal to or greater than a preset threshold value.

[0080] Furthermore, detection result determination unit 18 may determine the validity of the object detection result by comparing the types of detected objects. For example, suppose object detection unit 17 detects a truck based on the captured image, detects a light vehicle based on the first distance data, and detects a light vehicle based on the second distance data. In this case, object detection unit 17 determines that the validity of the object detection result based on the captured image is low, and that the validity of the object detection result based on the first distance data and the second distance data is high.

[0081] The detection result determination unit 18 adds information (hereinafter referred to as "validity determination result information") regarding whether the validity of the object detection result has been determined to be high, the validity of the object detection result has been determined to be low, or the object detection result has been determined to be undeterminable to the object detection result information output from the object detection unit 17, and outputs the information to the detection result correction unit 19.

[0082] Based on the validity determination result information attached to the object detection result information output from the detection result determination unit 18, the detection result correction unit 19 corrects the detection result of an object that the detection result determination unit 18 has determined to have low validity to the detection result of an object that the detection result determination unit 18 has determined to have high validity. To cite a specific example, suppose a person is detected in the object detection result information related to a captured image, and the validity determination result information attached to the object detection result information indicates low validity. Furthermore, suppose a car is detected in the object detection result information related to the first distance data and the second distance data, and the validity determination result information attached to the object detection result information indicates high validity. In this case, the detection result correction unit 19 corrects the information about the detected object in the object detection result information related to the captured image from information about the person to information about the car set in the object detection result information related to the first distance data and the second distance data. At this time, the detection result correction unit 19 adds information that can identify that the information about the detected object has been corrected to the object detection result information related to the captured image.

[0083] The detection result correction unit 19 outputs to the output unit 14 object detection result information that is deemed to have high validity and object detection result information that has been corrected to include information about a detected object that was deemed to have low validity. The detection result correction unit 19 stores in the noise DB 16 the object detection result information for which the object detection result is judged to be indeterminable.

[0084] The output unit 14 outputs the object detection result information output from the detection result correction unit 19. The output destination device to which the output unit 14 outputs the object detection result information is assumed to be determined in advance.

[0085] The operation of the sensor noise removal device 1a according to the second embodiment will be described. FIG. 8 is a flowchart for explaining the operation of the sensor noise removing device 1a according to the second embodiment. The sensor noise removal device 1a according to the second embodiment performs the operations described below with reference to the flowchart in Fig. 8 in addition to the operations of the sensor noise removal device 1 described in the first embodiment with reference to Fig. 4 and Fig. 5. Duplicate descriptions of the operations described in the first embodiment with reference to Fig. 4 and Fig. 5 will be omitted. The operations of steps ST402 to ST404 in FIG. 4 and the operations of steps ST801 to ST804 in FIG. 8 may be performed in parallel.

[0086] The object detection unit 17 acquires the sensor data acquired by the sensor data acquisition unit 11 (see step ST401 in FIG. 4), and detects an object for each piece of acquired sensor data (step ST801). The object detection unit 17 outputs object detection result information relating to the object detection result for each piece of sensor data to the detection result determination unit 18.

[0087] The detection result determination unit 18 determines the validity of the object detection result by the object detection unit 17 based on the object detection result information output from the object detection unit 17 in step ST801 (step ST802). The detection result determination unit 18 adds validity determination result information regarding whether the validity of the object detection result has been determined to be high, the validity of the object detection result has been determined to be low, or the object detection result has been determined to be undeterminable to the object detection result information output from the object detection unit 17, and outputs the result to the detection result correction unit 19.

[0088] The detection result correction unit 19 corrects the validity determination result attached to the object detection result information output from the detection result determination unit 18 in step ST802. information Based on this, the detection result determination unit 18 corrects the detection result of the object that it has determined to be of low validity to the detection result of the object that it has determined to be of high validity (step ST803). The detection result correction unit 19 outputs to the output unit 14 object detection result information that is deemed to have high validity and object detection result information that has been corrected to include information about a detected object that was deemed to have low validity. The detection result correction unit 19 stores in the noise DB 16 the object detection result information for which the object detection result is judged to be indeterminable.

[0089] The output unit 14 outputs the object detection result information output from the detection result correction unit 19 in step ST803 (step ST804).

[0090] In this way, the sensor noise removal device 1a detects an object for each of the acquired multiple pieces of sensor data and determines the validity of the object detection result. If the sensor noise removal device 1a determines that the validity of the object detection result is low, it corrects the object detection result determined to have low validity to an object detection result determined to have high validity. The sensor noise removal device 1a can detect errors in object detection by utilizing other sensor data.

[0091] In the second embodiment described above, the object detection unit 17 performs the object detection process on the sensor data acquired by the sensor data acquisition unit 11 and before the noise determination by the noise determination unit 12, but this is merely an example. For example, the object detection unit 17 may perform the object detection process on the sensor data determined as not having noise as a result of the noise determination by the noise determination unit 12, or may perform the object detection process on the sensor data output from the data replacement unit 13 after replacement by the data replacement unit 13.

[0092] The hardware configuration of the sensor noise removal device 1a according to the second embodiment is the same as the hardware configuration of the sensor noise removal device 1 described in the first embodiment with reference to FIGS. 6A and 6B, and therefore is not shown in the drawings. In the second embodiment, the functions of the sensor data acquisition unit 11, the noise determination unit 12, the data replacement unit 13, the output unit 14, the object detection unit 17, the detection result determination unit 18, and the detection result correction unit 19 are realized by the processing circuit 601. That is, the sensor noise removal device 1a includes the processing circuit 601 for controlling, when noise is present in the acquired sensor data, the processing circuit 601 to estimate noise-free sensor data for the noisy sensor data, generate replacement data corresponding to the noisy portion, replace the noise portion with the generated replacement data, and also to detect an object based on the sensor data and determine the validity of the detected object. The processing circuit 601 reads out and executes a program stored in the memory 605, thereby performing the functions of the sensor data acquisition unit 11, the noise determination unit 12, the data substitution unit 13, the output unit 14, the object detection unit 17, the detection result determination unit 18, and the detection result correction unit 19. That is, the sensor noise removal device 1a includes the memory 605 for storing a program that, when executed by the processing circuit 601, results in the execution of steps ST401 to ST404 in FIG. 4 and steps ST801 to ST804 in FIG. 8 described above. It can also be said that the program stored in the memory 605 causes a computer to execute the procedures or methods of the sensor data acquisition unit 11, the noise determination unit 12, the data substitution unit 13, the output unit 14, the object detection unit 17, the detection result determination unit 18, and the detection result correction unit 19. The sensor noise removal device 1a includes an input interface device 602 and an output interface device 603 that perform wired or wireless communication with a device such as a camera 21, a lidar 22, or a radar 23.

[0093] As described above, according to the second embodiment, the sensor noise removal device 1a includes an object detection unit 17 that detects an object for each of the plurality of sensor data acquired by the sensor data acquisition unit 11, a detection result determination unit 18 that determines the validity of the object detection result by the object detection unit 17, and a detection result correction unit 19 that corrects a detection result that the detection result determination unit 18 has determined to have low validity to a detection result that the detection result determination unit 18 has determined to have high validity. Therefore, the sensor noise removal device 1a can convert sensor data whose reliability has been reduced due to noise into sensor data in a noise-free state, and can detect errors in object detection by utilizing other sensor data.

[0094] Embodiment 3 In the first embodiment, the sensor noise removal device determines whether noise is present in the sensor data using a known technique. Furthermore, the sensor noise removal device performs replacement based on a predetermined rule in the first replacement function or the second replacement function. Specifically, for example, the sensor noise removal device generates replacement data for pixels included in a noise portion from neighboring pixels that are not affected by noise, and replaces the pixels in the noise portion with the generated replacement data in the first replacement function. Furthermore, for example, the sensor noise removal device infers, in the second replacement function, whether an object is detected in the noise portion from the first distance data or the second distance data that is not affected by noise, generates replacement data to indicate the object that is estimated to be detected in the noise portion based on the estimation result, and replaces the pixels in the noise portion with the generated replacement data. In the third embodiment, an embodiment will be described in which a sensor noise removal device performs noise determination and replacement based on a trained model in machine learning (hereinafter referred to as a "machine learning model").

[0095] Like the sensor noise removal device 1 according to the first embodiment, the sensor noise removal device 1b according to the third embodiment is mounted on a vehicle and connected to a camera 21, a lidar 22, and a radar 23. The sensor noise removal device 1b according to the third embodiment is further connected to a learning device 3. Details of the learning device 3 will be described later.

[0096] In embodiment 3, as in embodiment 1, the captured image obtained from camera 21 cannot be replaced with the first distance data obtained from lidar 22 or the second distance data obtained from radar 23 when processing is performed using the captured image. It is also assumed that an event that causes noise may occur in the camera 21. It is also assumed that an event that causes noise does not occur in the lidar 22 and the radar 23. In other words, it is assumed that no noise occurs in the first distance data and the second distance data.

[0097] FIG. 9 is a diagram illustrating an example of the configuration of a sensor noise removing device 1b according to the third embodiment. In the configuration of the sensor noise removal device 1b according to the third embodiment, the same components as those of the sensor noise removal device 1 described in the first embodiment with reference to FIG. 1 are assigned the same reference numerals, and redundant description will be omitted. The sensor noise removal device 1b according to the third embodiment differs from the sensor noise removal device 1 according to the first embodiment in that a model storage unit 30 is provided. Furthermore, the specific operations of the noise determination unit 12a and the data replacement unit 13a in the sensor noise removal device 1b of embodiment 3 are different from the specific operations of the noise determination unit 12 and the data replacement unit 13 in the sensor noise removal device 1 of embodiment 1.

[0098] The model storage unit 30 of the sensor noise removal device 1b stores a first machine learning model 301 and a second machine learning model 302. The second machine learning model 302 includes a machine learning model 3021 for a first substitution function and a machine learning model 3022 for a second substitution function. The first machine learning model 301 is a machine learning model that receives sensor data as input and outputs information indicating whether noise is present in the sensor data. The machine learning model 3021 for the first replacement function is a machine learning model that takes sensor data containing noise as input and outputs sensor data after the noise portion of the noisy sensor data has been replaced with sensor data that does not contain noise. The machine learning model 3022 for the second replacement function is a machine learning model that takes sensor data containing noise and sensor data without noise as input, and outputs sensor data after the noise portion of the sensor data containing noise has been replaced with sensor data without noise.

[0099] The first machine learning model 301 and the second machine learning model 302 stored in the model storage unit 30 are generated by the learning device 3. Details of the learning device 3 will be described later. 9, the model storage unit 30 is provided in the sensor noise removal device 1b, but this is merely an example. For example, the model storage unit 30 may be provided in a location outside the sensor noise removal device 1b that can be referenced by the sensor noise removal device 1b.

[0100] The noise determination unit 12a uses the first machine learning model 301 to determine whether or not noise has occurred in the sensor data acquired by the sensor data acquisition unit 11. Specifically, in the third embodiment, the noise determination unit 12a uses the first machine learning model 301 to determine whether or not noise has occurred in the captured image acquired by the sensor data acquisition unit 11.

[0101] The data replacement unit 13a uses the second machine learning model 302 to acquire sensor data in which the noise portion of the sensor data determined by the noise determination unit 12a to contain noise has been replaced with sensor data in which noise is not present. In this way, the data replacement unit 13a replaces the sensor data determined by the noise determination unit 12a to contain noise. In the third embodiment, the data replacement unit 13a acquires a captured image in which the noise portion of the captured image determined by the noise determination unit 12a to contain noise has been replaced with pixels in which noise is not present.

[0102] More specifically, the data replacement unit 13a receives the replacement possibility determination unit 131 and the noise determination unit 12. a When the information indicating that the sensor data is determined to contain noise by the noise determination unit 12a, in other words, that replacement is possible from only the captured image, the machine learning model for the first replacement function 3021 is used to obtain sensor data after the noise portion of the sensor data determined to contain noise by the noise determination unit 12a is replaced with sensor data that does not contain noise.

[0103] Furthermore, when the replacement possibility determination unit 131 outputs information indicating that replacement is possible based on sensor data determined by the noise determination unit 12 as not generating noise, in other words, the first distance data or the second distance data, the data replacement unit 13a uses the machine learning model 3022 for the second replacement function to acquire sensor data after the noise portion of the sensor data determined by the noise determination unit 12a as having noise has been replaced with sensor data without noise.

[0104] The operation of the sensor noise removal device 1b according to the third embodiment will be described later, and then an example of the configuration of the learning device 3 according to the third embodiment will be described. FIG. 10 is a diagram illustrating an example of the configuration of a learning device 3 according to the third embodiment. As shown in FIG. 9, the learning device 3 is connected to the sensor noise removal device 1b. The learning device 3 generates a first machine learning model 301 and a second machine learning model 302 through so-called supervised learning using training data. Specifically, the second machine learning model 302 is a machine learning model for a first substitution function 3021 and a machine learning model for a second substitution function 3022.

[0105] The learning device 3 includes a data acquisition unit 31 and a model generation unit 32 . The data acquisition unit 31 includes a first model data acquisition unit 311, a first substitution model data acquisition unit 312, and a second substitution model data acquisition unit 313. The model generation unit 32 includes a first model generation unit 321 , a first substitution model generation unit 322 , and a second substitution model generation unit 323 .

[0106] The data acquisition unit 31 acquires learning data. The first model data acquisition unit 311 of the data acquisition unit 31 acquires training data for generating the first machine learning model 301 (hereinafter referred to as "first model training data"). The first model learning data is data in which sensor data and truth labels are associated with each other. The truth labels are information indicating whether noise is present or not. The sensor data includes sensor data in which noise is present and sensor data in which noise is not present. A large amount of first model learning data is prepared in advance by a management company or the like.

[0107] The first substitution model data acquisition unit 312 of the data acquisition unit 31 acquires training data for generating the first substitution function machine learning model 3021 (hereinafter referred to as "first substitution model training data"). The first substitution model training data is data in which sensor data in which noise is generated is associated with a teacher label. Note that the noisy sensor data may include, for example, sensor data in which noise is generated due to the occurrence of an event that causes noise in the sensor, as well as sensor data in which noise is generated due to the influence of the environment, etc. The teacher label is sensor data generated by treating the noisy part of the associated sensor data as if it were in a noise-free state. A large amount of first substitution model training data is prepared in advance by a management company, etc.

[0108] The second substitution model data acquisition unit 313 of the data acquisition unit 31 acquires training data for generating the machine learning model 3022 for the second substitution function (hereinafter referred to as "second substitution model training data"). The second substitution model training data is data in which noisy sensor data, noise-free sensor data different from the noisy sensor data, and teacher labels are associated with each other. The teacher labels are sensor data generated by replacing the noisy part of the noisy sensor data with a noise-free state. A large amount of second substitution model training data is prepared in advance by a management company or the like. Note that the noisy sensor data and the noise-free sensor data are sensor data acquired for the same detection range under the same conditions.

[0109] The data acquiring unit 31 outputs the acquired learning data to the model generating unit 32. Specifically, the data acquiring unit 31 outputs the first model learning data acquired by the first model data acquiring unit 311, the first replacement model learning data acquired by the first replacement model data acquiring unit 312, and the second replacement model learning data acquired by the second replacement model data acquiring unit 313 to the model generating unit 32. In addition, for each of the first model learning data, the first substitution model learning data, and the second substitution model learning data, the data acquisition unit 31 is configured to determine, depending on the type of sensor data contained in the learning data, which type of sensor data the learning data was generated in accordance with.

[0110] The model generation unit 32 generates a first machine learning model 301, a first substitution function machine learning model 3021, and a second substitution function machine learning model 3022. The first model generation unit 321 of the model generation unit 32 uses a neural network to input the first model learning data output from the data acquisition unit 31 and generate a first machine learning model 301 that outputs information on whether or not noise is occurring. When generating the first machine learning model 301, the first model generation unit 321 performs preprocessing such as feature extraction on the first model learning data. Specifically, for example, if the sensor data is a captured image, the first model generation unit 321 divides the image into images in units of one pixel. Furthermore, for example, the first model generation unit 321 attaches a label such as "object detection detected." Note that this preprocessing may be performed by the first model data acquisition unit 311, which may then output the preprocessed data to the model generation unit 32 as learning data.

[0111] A neural network consists of an input layer consisting of multiple neurons, an intermediate layer (hidden layer) consisting of multiple neurons, and an output layer consisting of multiple neurons. The intermediate layer may be one layer or two or more layers. FIG. 11 is a diagram illustrating an example of a neural network. For example, in a three-layer neural network as shown in Figure 11, when multiple inputs are input to the input layer (X1-X3), the values ​​are multiplied by weights W1 (w11-w16) and sent to the middle layer. ( The result is multiplied by a weight W2 (w21-w26) and output from the output layer (Z1-Z3). The output result changes depending on the values ​​of the weights W1 and W2.

[0112] In the third embodiment, first model generation unit 321 causes first machine learning model 301, which is configured as a neural network as described above, to learn using so-called supervised learning based on first model learning data. The first machine learning model 301 learns by adjusting the weights W1 and W2 so that more correct answers are output from the output layer. The first model generation unit 321 generates the first machine learning model 301 as described above, and outputs it to the model storage unit 30 (see FIG. 9).

[0113] The first model generation unit 321 generates the first machine learning model 301 according to the type of sensor data included in the first model learning data, and makes it clear which type of sensor data the generated first machine learning model 301 is based on.

[0114] The first substitution model generation unit 322 uses a neural network to input the first substitution model learning data output from the data acquisition unit 31, and generates a machine learning model 3021 for the first substitution function that outputs sensor data after the noisy parts of the sensor data in which noise is generated have been replaced with sensor data in which noise is not generated. When generating the first replacement function machine learning model 3021, the first replacement model generation unit 322 performs preprocessing such as feature extraction on the first replacement model training data. Specifically, for example, if the sensor data is a captured image, the first replacement model generation unit 322 divides the image into images in units of one pixel. Furthermore, for example, the first replacement model generation unit 322 attaches a label such as "object detection detected." Note that this preprocessing may be performed by the first replacement model data acquisition unit 312, which may then output the preprocessed data to the model generation unit 32 as training data.

[0115] In embodiment 3, the first substitution model generation unit 322 trains the machine learning model for the first substitution function 3021, which is configured as a neural network as described above (see Figure 11), using so-called supervised learning based on the first substitution model training data. The first replacement function machine learning model 3021 learns by adjusting the weights W1 and W2 so that more correct answers are output from the output layer.

[0116] The machine learning model for first substitution function 3021 is designed to treat noise occurring in sensor data as sensor data in a state where the noise is not occurring. Specifically, for example, assume that noise occurs in an image captured by camera 21. The machine learning model for first substitution function 3021 receives an image in which noise is occurring as an input, and outputs an image in a state where the noise is not occurring. The first substitution model generation unit 322 generates the first substitution function machine learning model 3021 as described above, and outputs it to the model storage unit 30 (see FIG. 9).

[0117] The first substitution model generation unit 322 generates the machine learning model 3021 for the first substitution function according to the type of sensor data containing noise contained in the first substitution model learning data, and makes it clear which type of sensor data the generated machine learning model 3021 for the first substitution function was generated in accordance with.

[0118] The second substitution model generation unit 323 uses a neural network to input the second substitution model learning data output from the data acquisition unit 31, and generates a machine learning model 3022 for the second substitution function that outputs sensor data after the noisy parts of the noisy sensor data have been replaced with sensor data that does not contain noise. When generating the machine learning model for second replacement function 3022, the second replacement model generation unit 323 performs preprocessing such as feature extraction on the second replacement model training data. Specifically, for example, if the sensor data is a captured image, the second replacement model generation unit 323 divides the image into images in units of one pixel. Furthermore, for example, the second replacement model generation unit 323 attaches a label such as "object detection detected." Note that this preprocessing may be performed by the data acquisition unit for second replacement model 313, which may then output the preprocessed data to the model generation unit 32 as training data.

[0119] In embodiment 3, the second substitution model generation unit 323 trains the machine learning model for the second substitution function 3022, which is configured as a neural network as described above (see Figure 11), using so-called supervised learning based on the second substitution model training data. The second replacement function machine learning model 3022 learns by adjusting the weights W1 and W2 so that more correct answers are output from the output layer.

[0120] The machine learning model 3022 for the second substitution function is designed to convert noise occurring in sensor data into sensor data in a state where the noise is not occurring, based on other sensor data. Specifically, for example, the sensor data is assumed to be an image captured by the camera 21, first distance data acquired by the lidar 22, and second distance data acquired by the radar 23. Of these, it is assumed that noise occurs in the captured image. No noise occurs in the first distance data and the second distance data. In this case, the machine learning model 3022 for the second substitution function receives as input the captured image in which noise occurs and the first distance data and second distance data in which noise is not occurring, and outputs a captured image in a state in which noise is not occurring. The second substitution model generation unit 323 generates the machine learning model for the second substitution function 3022 as described above, and outputs it to the model storage unit 30 (see FIG. 9).

[0121] The second substitution model generation unit 323 generates the machine learning model 3022 for the second substitution function according to the type of sensor data containing noise contained in the second substitution model learning data, and makes it clear which type of sensor data the generated machine learning model 3022 for the second substitution function was generated in accordance with.

[0122] The operation of the sensor noise removal device 1b according to the third embodiment will be described. FIG. 12 is a flowchart for explaining the operation of the sensor noise removing device 1b according to the third embodiment. The sensor data acquisition unit 11 acquires sensor data relating to the surrounding conditions of the vehicle (step ST1201). Specifically, the sensor data acquisition unit 11 acquires an image captured by the camera 21, first distance data acquired by the lidar 22, and second distance data acquired by the radar 23. The sensor data acquisition unit 11 outputs the acquired captured image, first distance data, and second distance data to the noise determination unit 12. The sensor data acquiring unit 11 also stores the acquired captured image, first distance data, and second distance data in the sensor DB 15.

[0123] The noise determining unit 12a determines whether or not noise occurs in the sensor data acquired by the sensor data acquiring unit 11 in step ST1201 (step ST1202). Specifically, the noise determination unit 12a uses the first machine learning model 301 to determine whether or not noise has occurred in the sensor data acquired by the sensor data acquisition unit 11. In the third embodiment, the noise determination unit 12a uses the first machine learning model 301 to determine whether or not noise has occurred in the captured image acquired by the sensor data acquisition unit 11. The noise determination unit 12a outputs the captured image acquired from the sensor data acquisition unit 11 to the data replacement unit 13a together with the determination result as to whether or not the captured image contains noise. At this time, the noise determination unit 12a also outputs the first distance data and the second distance data acquired from the sensor data acquisition unit 11 to the data replacement unit 13a.

[0124] The data replacing unit 13a replaces the sensor data determined by the noise determining unit 12a in step ST1202 as having noise with sensor data in a state where no noise is occurring (step ST1203). Specifically, the data replacement unit 13a uses the second machine learning model 302 to replace the noise portion of the sensor data determined by the noise determination unit 12a as having noise with sensor data without noise, and acquires the sensor data. a For a captured image determined to contain noise by the method described above, the noise portion is replaced with pixels without noise, and a captured image is obtained.

[0125] More specifically, when the replacement possibility determination unit 131 outputs information indicating that replacement is possible from sensor data determined by the noise determination unit 12a to contain noise, in other words, from the captured image alone, the data replacement unit 13a uses the machine learning model 3021 for the first replacement function to obtain a replaced captured image in which the noise portion of the captured image determined by the noise determination unit 12a to contain noise has been replaced with pixels that do not contain noise. Furthermore, when the replacement possibility determination unit 131 outputs information indicating that replacement is possible based on sensor data determined by the noise determination unit 12a as not generating noise, in other words, the first distance data or the second distance data, the data replacement unit 13a uses the machine learning model 3022 for the second replacement function to obtain a replaced captured image in which the noise portion of the captured image determined by the noise determination unit 12a as having noise has been replaced with pixels that do not generate noise.

[0126] When the data substitution unit 13a has performed substitution on the captured image, it outputs the substituted captured image to the output unit 14. When the data substitution unit 13a has not performed substitution on the captured image, it outputs the captured image acquired by the sensor data acquisition unit 11 to the output unit 14. In addition, the data substitution unit 13 a outputs the first distance data and the second distance data acquired by the sensor data acquisition unit 11 to the output unit 14.

[0127] The output unit 14 outputs the sensor data output from the data replacement unit 13a in step ST1203 (step ST1204). a outputs the replaced captured image or the captured image output from the data replacing unit 13, the first distance data, and the second distance data.

[0128] The operation of the learning device 3 according to the third embodiment will be described. FIG. 13 is a flowchart illustrating the operation of the learning device 3 according to the third embodiment.

[0129] The data acquisition unit 31 acquires learning data (step ST1301). The first model data acquisition unit 311 of the data acquisition unit 31 acquires first model training data. The first substitution model data acquisition unit 312 of the data acquisition unit 31 acquires first substitution model training data. The second substitution model data acquisition unit 313 of the data acquisition unit 31 acquires second substitution model training data. The data acquisition unit 31 outputs the acquired learning data to the model generation unit 32.

[0130] The model generation unit 32 generates the first machine learning model 301, the first substitution function machine learning model 3021, and the second substitution function machine learning model 3022 (step ST1302). Specifically, the first model generation unit 321 of the model generation unit 32 receives the first model learning data output from the data acquisition unit 31 in step ST1301 as input, and generates a first machine learning model 301 that outputs information on whether or not noise is occurring. The first model generation unit 321 outputs the generated first machine learning model 301 to the model storage unit 30. The first substitution model generation unit 322 of the model generation unit 32 receives as input the first substitution model learning data output from the data acquisition unit 31 in step ST1301, and generates a first substitution function machine learning model 3021 that outputs sensor data after the noisy portion of the noisy sensor data has been replaced with noise-free sensor data. The first substitution model generation unit 322 outputs the generated first substitution function machine learning model 3021 to the model storage unit 30. The second substitution model generation unit 323 of the model generation unit 32 receives as input the second substitution model learning data output from the data acquisition unit 31 in step ST1301, and generates a machine learning model for second substitution function 3022 that outputs sensor data after the noisy portion of the noisy sensor data has been replaced with sensor data without noise. The second substitution model generation unit 323 outputs the generated machine learning model for second substitution function 3022 to the model storage unit 30.

[0131] The hardware configuration of the sensor noise removal device 1b according to the third embodiment is the same as the hardware configuration of the sensor noise removal device 1 described in the first embodiment with reference to FIGS. 6A and 6B, and therefore is not shown in the drawings. In the third embodiment, the functions of the sensor data acquisition unit 11, the noise determination unit 12a, the data replacement unit 13a, and the output unit 14 are realized by the processing circuit 601. That is, when noise occurs in the acquired sensor data, the sensor noise removal device 1b includes the processing circuit 601 for performing control to acquire noise-free sensor data for the noisy sensor data using the first machine learning model 301, the first replacement function machine learning model 3021, or the second replacement function machine learning model 3022. The processing circuit 601 reads out and executes a program stored in the memory 605, thereby performing the functions of the sensor data acquisition unit 11, the noise determination unit 12a, the data substitution unit 13a, and the output unit 14. That is, the sensor noise removal device 1b includes the memory 605 for storing a program that, when executed by the processing circuit 601, results in the execution of steps ST1201 to ST1204 in FIG. 12 described above. It can also be said that the program stored in the memory 605 causes a computer to execute the procedures or methods of the sensor data acquisition unit 11, the noise determination unit 12a, the data substitution unit 13a, and the output unit 14. The sensor DB 15, the noise DB 16, and the model storage unit 30 use the memory 605. Note that this is just an example, and the sensor DB 15 and the noise DB 16 may be configured with an HDD, an SSD (Solid State Drive), a DVD, or the like. The sensor noise removal device 1b includes an input interface device 602 and an output interface device 603 that perform wired or wireless communication with devices such as the camera 21, the lidar 22, the radar 23, or the learning device 3.

[0132] The learning device 3 according to the third embodiment has the same hardware configuration as the sensor noise removal device 1 according to the first embodiment (see FIGS. 6A and 6B). In the third embodiment, the functions of the data acquisition unit 31 and the model generation unit 32 are realized by a processing circuit 601. That is, the learning device 3 includes a processing circuit 601 for generating a first machine learning model 301, a first replacement function machine learning model 3021, and a second replacement function machine learning model 3022 based on the acquired learning data. The processing circuit 601 may be dedicated hardware as shown in FIG. 6A, or may be a CPU (Central Processing Unit) 604 that executes a program stored in a memory 605 as shown in FIG. 6B.

[0133] When the processing circuitry 601 is dedicated hardware, the processing circuitry 601 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

[0134] When the processing circuit 601 is a CPU 604, the functions of the data acquisition unit 31 and the model generation unit 32 are realized by software, firmware, or a combination of software and firmware. The software or firmware is written as a program and stored in the memory 605. The processing circuit 601 executes the functions of the data acquisition unit 31 and the model generation unit 32 by reading and executing the program stored in the memory 605. That is, the learning device 3 includes the memory 605 for storing a program that, when executed by the processing circuit 601, results in the execution of steps ST1301 to ST1302 in FIG. 13 described above. It can also be said that the program stored in the memory 605 causes a computer to execute the procedures or methods of the data acquisition unit 31 and the model generation unit 32. Here, memory 605 refers to, for example, non-volatile or volatile semiconductor memory such as RAM, ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), or magnetic disk, flexible disk, optical disk, compact disk, mini disk, DVD (Digital Versatile Disc), etc.

[0135] It is also possible to realize some of the functions of the data acquisition unit 31 and the model generation unit 32 with dedicated hardware and some with software or firmware. For example, the functions of the data acquisition unit 31 can be realized by a processing circuit 601 as dedicated hardware, and the functions of the model generation unit 32 can be realized by the processing circuit 601 reading and executing a program stored in a memory 605. The learning device 3 also includes an input interface device 602 and an output interface device 603 that perform wired or wireless communication with devices such as the sensor noise removal device 1b.

[0136] In the third embodiment described above, the learning device 3 is provided outside the sensor noise removal device 1b and is connected to the sensor noise removal device 1b via a network, but this is merely an example. The learning device 3 may be provided in the sensor noise removal device 1b.

[0137] Furthermore, in the above third embodiment, the data replacement unit 13a has the function of acquiring noise-free sensor data using the machine learning model for first replacement function 3021 and the function of acquiring noise-free sensor data using the machine learning model for second replacement function 3022, but this is merely an example. The data replacement unit 13a may have either the function of acquiring noise-free sensor data using the machine learning model for first replacement function 3021 or the function of acquiring noise-free sensor data using the machine learning model for second replacement function 3022. If the data replacement unit 13a only has the function of acquiring noise-free sensor data using the first replacement function machine learning model 3021, the replacement feasibility determination unit 131 only determines whether the first replacement condition is satisfied. In this case, the learning device 3 does not necessarily need to generate the second replacement function machine learning model 3022. Furthermore, if the data replacement unit 13a only has the function of acquiring noise-free sensor data using the second replacement function machine learning model 3022, the replacement feasibility determination unit 131 only determines whether the second replacement condition is satisfied. In this case, it is not essential for the learning device 3 to generate the first replacement function machine learning model 3021.

[0138] Furthermore, the above embodiments 3 In the above description, the data replacement unit 13a is provided with the replacement possibility determination unit 131, but the replacement possibility determination unit 131 is not essential. For example, the data replacement unit 13a may have the function of the replacement possibility determination unit 131, and the data replacement unit 13a may determine whether or not the replacement possibility condition is satisfied when performing replacement.

[0139] In the third embodiment, it is assumed that noise may occur in the captured image, but this is merely an example. In the third embodiment, it may also be assumed that noise may occur in the first distance data and the second distance data. The noise determination unit 12a can determine whether or not noise is occurring in all of the sensor data acquired by the sensor data acquisition unit 11. For example, the noise determining unit 12a can use the first machine learning model 301 to determine whether noise occurs in the first distance data or the second distance data.

[0140] As described above, according to the third embodiment, the sensor noise removal device 1b is configured to include: a sensor data acquisition unit 11 that acquires sensor data related to the surrounding conditions of the vehicle; a noise determination unit 12a that determines whether noise is present in the sensor data acquired by the sensor data acquisition unit 11 using a first machine learning model 301 that receives the sensor data as an input and outputs information indicating whether noise is present in the sensor data; and a data replacement unit 13a that acquires sensor data after replacing the noise portion of the sensor data determined by the noise determination unit 12a to contain noise with sensor data in a noise-free state using a second machine learning model 302. Therefore, the sensor noise removal device 1b can convert sensor data whose reliability has been reduced due to noise into sensor data in a noise-free state.

[0141] In the above first to third embodiments, the camera 21, the lidar 22, and the radar 23 are assumed to be mounted on a vehicle, and the noise-free sensor data used for replacement is the sensor data acquired from the lidar 22 or the radar 23 mounted on the vehicle. However, this is merely an example. For example, in the above-described first to third embodiments, the noise-free sensor data used for the replacement may be acquired from another vehicle other than the subject vehicle, the cloud, or a device installed on the road.

[0142] Furthermore, in the above first to third embodiments, it is assumed that only one sensor of the same type is provided, but this is merely an example. For example, a plurality of sensors of the same type may be mounted on a vehicle. As a specific example, two cameras 21, a lidar 22, and a radar 23 may be mounted on a vehicle, and the sensor noise removal devices 1, 1a, and 1b may acquire sensor data from the two cameras 21, the lidar 22, and the radar 23. In this case, when replacing sensor data containing noise with sensor data containing no noise, the sensor noise removal devices 1, 1a, and 1b preferentially use sensor data of the same type. For example, if a captured image acquired from one camera 21 contains noise and a captured image acquired from the other camera 21 does not contain noise, the sensor noise removal devices 1, 1a, and 1b replace the noisy portion of the captured image acquired from one camera 21 with the captured image acquired from the other camera 21.

[0143] Furthermore, in the above first to third embodiments, the sensor noise removal devices 1, 1a, 1b are assumed to be on-board devices mounted on a vehicle, and the sensor data acquisition unit 11, the noise determination unit 12, 12a, the data substitution unit 13, 13a, and the output unit 14 are assumed to be provided in the sensor noise removal devices 1, 1a, 1b. However, without being limited to this, some of the sensor data acquisition unit 11, the noise determination unit 12, 12a, the data substitution unit 13, 13a, and the output unit 14 may be mounted in the on-board device of the vehicle, and the rest may be provided in a server connected to the on-board device via a network, and the on-board device and the server may form a sensor noise removal system. For example, the noise determination unit 12, 12a and the data replacement unit 13, 13a may be provided in a server, and the sensor data acquisition unit 11 and the output unit 14 may be provided in an in-vehicle device. The noise determination unit 12, 12a acquires sensor data from the in-vehicle device. The data replacement unit 13, 13a outputs the replaced sensor data to the in-vehicle device.

[0144] In addition, the present disclosure allows for free combination of the respective embodiments, modification of any of the components of the respective embodiments, or omission of any of the components of the respective embodiments.

[0145] The sensor noise removal device according to the present disclosure is configured to be able to convert sensor data whose reliability has been reduced due to noise into sensor data in a noise-free state, and can therefore be applied to sensor noise removal devices mounted on vehicles, etc. that perform processing using sensor data. [Explanation of symbols]

[0146] 1a, 1b Sensor noise removal device, 21 Camera, 22 Lidar, 23 Radar, 11 Sensor data acquisition unit, 12, 12a Noise determination unit, 13a Data replacement unit, 131 Replacement possibility determination unit, 14 Output unit, 15 Sensor DB, 16 Noise DB, 17 Object detection unit, 18 Detection result determination unit, 19 Detection result correction unit, 30 Model storage unit, 301 First machine learning model, 302 Second machine learning model, 3021 Machine learning model for first replacement function, 3022 Machine learning model for second replacement function, 3 Learning device, 31 Data acquisition unit, 311 Data acquisition unit for first model, 312 Data acquisition unit for first replacement model, 313 Data acquisition unit for second replacement model, 32 Model generation unit, 321 First model generation unit, 322 First replacement model generation unit, 323 Second replacement model generation unit, 601 Processing circuit, 602 An input interface device, 603 an output interface device, 604 a CPU, and 605 a memory.

Claims

1. a sensor data acquisition unit that acquires a plurality of sensor data relating to a surrounding situation of the vehicle, the sensor data being acquired by a plurality of sensors of different types; a noise determination unit that determines whether noise is occurring in the plurality of pieces of sensor data; a data replacement unit that, when there is first sensor data in which the noise determination unit has determined that the noise is occurring, and second sensor data acquired by a sensor of a type different from the sensor that acquired the first sensor data and in which the noise determination unit has determined that the noise is not occurring, and when an object existing in real space corresponding to a noise portion of the first sensor data is not detected in the second sensor data, replaces the first sensor data in which the noise is occurring with first sensor data in which the noise is not occurring; A sensor noise removal device comprising:

2. An object detection unit that detects the object for each of the plurality of sensor data acquired by the sensor data acquisition unit; a detection result determination unit that determines the validity of the object detection result by the object detection unit; a detection result correction unit that corrects the detection result so as not to use the detection result of the object that the detection result determination unit has determined to have low validity; The sensor noise removal device according to claim 1 , comprising:

3. The plurality of sensors includes a first sensor that outputs the first sensor data and a second sensor that outputs the second sensor data; The first sensor is a lidar and the second sensor is a radar.

2. The sensor noise removal device according to claim 1.

4. The noise determination unit determines whether the noise occurs in the first sensor data based on a characteristic of the first sensor data.

2. The sensor noise removal device according to claim 1.

5. a step in which a sensor data acquisition unit acquires a plurality of sensor data relating to a surrounding situation of the vehicle, the sensor data being acquired by a plurality of sensors of different types; a step in which a noise determination unit determines whether noise is occurring in the plurality of pieces of sensor data; a data replacement unit, when there is first sensor data in which the noise determination unit has determined that the noise is occurring, and second sensor data acquired by a sensor of a type different from that of the sensor that acquired the first sensor data and in which the noise determination unit has determined that the noise is not occurring, and when an object existing in real space corresponding to a noise portion of the first sensor data is not detected in the second sensor data, replacing the first sensor data in which the noise is occurring with first sensor data in which the noise is not occurring; A sensor noise removal method comprising:

6. A step in which an object detection unit detects the object for each of the plurality of sensor data acquired by the sensor data acquisition unit; a step in which a detection result determination unit determines the validity of a detection result of the object by the object detection unit; a detection result correction unit correcting the detection result so as not to use the detection result of the object determined by the detection result determination unit to have low validity; The sensor noise removal method according to claim 5, further comprising:

Images