Object recognition device and object recognition method
The object recognition device dynamically adjusts neural network sparseness ratios based on driving conditions to maintain accuracy and speed up object recognition in varying vehicle environments.
Patent Information
- Application Number
- JP2025078451
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2025-05-09
- Publication Date
- 2026-11-19
AI Technical Summary
Existing object recognition techniques in vehicles reduce accuracy when sparsifying neural networks for faster inference, particularly in varying driving environments.
An object recognition device that dynamically adjusts the sparseness ratio of neural network intermediate layers based on the vehicle's driving environment, using a switching mechanism to maintain accuracy and reduce recognition time.
The device achieves faster object recognition while preserving accuracy by adaptively adjusting the sparseness ratio of neural network layers, mitigating accuracy loss due to environmental changes.
Smart Images

Figure 2026182313000001_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to an object recognition device and an object recognition method.
Background Art
[0002] A technique is known that shortens the inference calculation time of a neural network by sparsifying the neural network (for example, Patent Document 1).
Prior Art Documents
Patent Documents
[0003]
Patent Document 1
Summary of the Invention
Problems to be Solved by the Invention
[0004] When recognizing an object in an image acquired from a camera mounted on a vehicle, the time required for object recognition can be shortened by sparsifying the neural network. However, depending on the driving environment of the vehicle, sparsification may reduce the accuracy of object recognition. Accordingly, there is a demand for a technique that reduces the time required for object recognition while maintaining the accuracy of object recognition.
Means for Solving the Problems
[0005] The present disclosure can be implemented in the following forms.
[0006] According to one embodiment of the present disclosure, an object recognition device (100) is provided. This object recognition device includes an acquisition unit (105) that acquires an image from a camera (20) mounted on a vehicle (10), a driving environment determination unit (120) that determines the driving environment of the vehicle, a switching unit (140) that switches the sparseness ratio of each intermediate layer of a neural network (ML) having a plurality of intermediate layers (Lm) according to the result of the driving environment determination, and a recognition unit (150) that recognizes an object in the image using the neural network.
[0007] This type of object recognition device allows for switching the sparseness ratio of each intermediate layer according to the driving environment, thereby reducing the time required for object recognition while maintaining the accuracy of object recognition. [Brief explanation of the drawing]
[0008] [Figure 1] An explanatory diagram showing the configuration of a vehicle equipped with an object recognition device. [Figure 2] An explanatory diagram showing the functional configuration of an object recognition device. [Figure 3] A schematic diagram illustrating the structure of the recognition model. [Figure 4] An explanatory diagram showing the correspondence between the driving environment and weight data. [Figure 5] A flowchart showing the procedure for creating weight data. [Figure 6] A schematic diagram illustrating the method for creating weight data. [Figure 7] A flowchart illustrating the steps involved in object recognition. [Figure 8] A schematic diagram illustrating how weight data is switched according to the driving environment. [Modes for carrying out the invention]
[0009] A. Embodiments: As shown in Figure 1, the object recognition device 100 in one embodiment of the present disclosure is mounted on a vehicle 10. In addition to the object recognition device 100, the vehicle 10 is equipped with a camera 20, a solar radiation sensor 30, a rain sensor 40, a vehicle speed sensor 50, an ETC onboard unit 60, a navigation device 70, and a vehicle control device 80. The object recognition device 100 recognizes objects in an image using an image acquired from the camera 20 and a recognition model ML.
[0010] The object recognition device 100 is configured with a computer. In this embodiment, the object recognition device 100 includes an ISP (Image Signal Processor) 101, a CPU (Central Processing Unit) 102, an NPU (Neural Network Processing Unit) 103, a memory 104, an input / output interface 105, and an internal bus 106. The ISP 101, CPU 102, NPU 103, memory 104, and input / output interface 105 are connected via the internal bus 106 to enable bidirectional communication. The memory 104 stores a recognition model ML used for object recognition and a plurality of weight data WD. A camera 20, a solar radiation sensor 30, a rain sensor 40, a vehicle speed sensor 50, an ETC onboard unit 60, a navigation device 70, and a vehicle control device 80 are connected to the input / output interface 105. In this disclosure, the input / output interface 105 may be referred to as the acquisition unit.
[0011] As shown in Figure 2, the object recognition device 100 includes the functions of an image processing unit 110, a driving environment determination unit 120, a decision unit 130, a switching unit 140, and a recognition unit 150. In this embodiment, the image processing unit 110 is realized by the ISP 101 executing a computer program stored in the memory 104. The driving environment determination unit 120 and the decision unit 130 are realized by the CPU 102 executing a computer program stored in the memory 104. The switching unit 140 and the recognition unit 150 are realized by the NPU 103 executing a computer program stored in the memory 104.
[0012] The image processing unit 110 performs image processing on the image acquired from the camera 20. In this embodiment, the raw image data is input to the image processing unit 110. Image processing by the image processing unit 110 includes, for example, adjustments to brightness, contrast, and color balance.
[0013] The driving environment determination unit 120 determines the driving environment of the vehicle 10. In this embodiment, the driving environment determination unit 120 includes a day / night determination unit 121, a sunny / rainy day determination unit 122, a road determination unit 123, and a traffic congestion determination unit 124.
[0014] The day / night determination unit 121 determines whether the driving environment of the vehicle 10 is daytime or nighttime. In this embodiment, the day / night determination unit 121 uses a solar radiation sensor 30 that detects the amount of solar radiation to determine whether it is daytime or nighttime. Alternatively, the day / night determination unit 121 may use the time obtained by the real-time clock function of the CPU 102 to determine whether it is daytime or nighttime.
[0015] The weather determination unit 122 determines whether the driving environment of the vehicle 10 is sunny or rainy. In this embodiment, the weather determination unit 122 uses a rain sensor 40 to detect rain and determines whether it is sunny or rainy. Alternatively, the weather determination unit 122 may use weather information obtained from a weather server (not shown) located outside the vehicle 10 to determine whether it is sunny or rainy.
[0016] The road determination unit 123 determines whether the traveling environment of the vehicle 10 is an expressway or a general road. In the present embodiment, the road determination unit 123 uses at least one of the vehicle speed sensor 50, the ETC on-board unit 60, and the navigation device 70 to determine whether the road is an expressway or a general road. The vehicle speed sensor 50 detects the traveling speed of the vehicle 10. The ETC on-board unit 60 outputs a signal indicating that the vehicle 10 has entered an expressway and a signal indicating that the vehicle 10 has exited the expressway. The navigation device 70 includes a GPS receiver and an IMU, and can acquire position information of the vehicle 10 using the GPS receiver and the IMU. Map information is stored in the navigation device 70. The map information indicates the position of the expressway. The road determination unit 123 uses at least one of the detection result of the vehicle speed sensor 50, the output signal of the ETC on-board unit 60, and the position information and map information of the navigation device 70 to determine whether the road is an expressway or a general road.
[0017] The congestion status determination unit 124 determines whether the traveling environment of the vehicle 10 is in a traffic jam or not in a traffic jam. In the present embodiment, the congestion status determination unit 124 uses at least one of the vehicle speed sensor 50, the navigation device 70, and the vehicle control device 80 to determine whether the vehicle is in a traffic jam or not in a traffic jam. The navigation device 70 can acquire congestion information from a road traffic information communication system (not shown) located outside the vehicle 10. The vehicle control device 80 executes control related to acceleration, braking, and steering of the vehicle 10. The congestion status determination unit 124 uses at least one of the detection result of the vehicle speed sensor 50, the position information and congestion information of the navigation device 70, and the control signal of the vehicle control device 80 to determine whether the vehicle is in a traffic jam or not in a traffic jam.
[0018] The determination unit 130 determines weight data WD used for object recognition by the recognition model ML according to the driving environment determination result from the driving environment determination unit 120. In the present embodiment, the determination unit 130 determines the weight data WD to be used for object recognition by the recognition model ML from among the plurality of pieces of weight data WD stored in the memory 104, according to the determination results of the day / night determination unit 121, the weather determination unit 122, the road determination unit 123, and the traffic jam situation determination unit 124.
[0019] The switching unit 140 switches the weight data WD used for object recognition by the recognition model ML. In the present embodiment, the switching unit 140 acquires the weight data WD determined by the determination unit 130 from the memory 104, and switches the weight data WD used for object recognition by the recognition model ML.
[0020] The recognition unit 150 executes object recognition using the recognition model ML and the weight data WD. In the present embodiment, the recognition unit 150 uses the recognition model ML and the weight data WD to recognize an object in an image acquired from the camera 20 via the image processing unit 110.
[0021] Figure 3 schematically shows the configuration of the recognition model ML. In this embodiment, the recognition model ML is composed of a neural network. In this disclosure, the term "neural network" refers to an artificial neural network, not a biological nerve cell. The recognition model ML comprises an input layer Li, a plurality of hidden layers Lm1 to Lm5, and an output layer Lo. In the following description, when each hidden layer Lm1 to Lm5 is described without particular distinction, it will simply be referred to as hidden layer Lm. Although five hidden layers Lm1 to Lm5 are illustrated in Figure 3, the number of hidden layers Lm in the recognition model ML is not limited to five. Each layer Li, Lm, and Lo of the recognition model ML comprises at least one node that simulates a neuron. Nodes in adjacent layers are connected by junctions that simulate synapses. Each junction is assigned a weight that represents the strength of the connection between nodes. The ML recognition model is pre-trained using supervised learning with a training dataset containing training images and ground truth labels. When an image is input to the input layer Li, each hidden layer Lm performs a sum-of-products operation using weights, and the recognition result of the object in the image is output from the output layer Lo. For example, the backpropagation method is used as the training algorithm for the ML recognition model. In backpropagation, the weights are adjusted to minimize the error between the ground truth and the output.
[0022] Figure 4 shows the correspondence between the driving environment determination result by the driving environment determination unit 120 and the weight data WD. In this embodiment, the driving environment determination unit 120 outputs 16 different determination results. Memory 104 stores 16 weight data WD1 to WD16 that are associated one-to-one with the 16 different determination results. Each of the weight data WD1 to WD16 has a different sparseness ratio for each intermediate layer Lm. In this disclosure, the sparseness ratio means the proportion of invalid weights among the multiple weights used in the sum-of-products operation of the intermediate layer Lm. An invalid weight includes a weight that is zero. For example, if 90% of the multiple weights used in the sum-of-products operation of a certain intermediate layer Lm are zero, the sparseness ratio of this intermediate layer Lm is 90%. Each of the weight data WD1 to WD16 is configured to invalidate a portion of the weights that are included in the rank according to the sparseness ratio when the multiple weights for each intermediate layer Lm are ranked from the smallest absolute value of the weights. The higher the sparseness ratio of the intermediate layer Lm, the shorter the calculation time for sum-of-products operations in the intermediate layer Lm. In this embodiment, among the 16 weight data WD1 to WD16, the weight data WD corresponding to the determination result of the driving environment determination unit 120 is used for object recognition by the recognition model ML.
[0023] Figure 5 is a flowchart showing the procedure for creating weight data. Here, we will explain how to create weight data WD1. Weight data WD1 is used for object recognition by the recognition model ML when the driving environment is "daytime", "sunny", "highway", and "not congested". The other 15 weight data sets WD2 to WD16 are created using the same method.
[0024] In step S11, the sparseness ratio of each hidden layer Lm of the recognition model ML is increased. Here, the sparseness ratio of each hidden layer Lm is increased compared to the sparseness ratio of each hidden layer Lm immediately after training. As shown in Figure 6, for example, the sparseness ratio of each hidden layer Lm1 to Lm5 is changed from 0% to 90%. In this embodiment, the sparseness ratio of each hidden layer Lm is increased by disabling some of the weights used in the sum-of-products operation of each hidden layer Lm. Disabling weights is also called pruning. Specifically, of the weights used in the sum-of-products operation of each hidden layer Lm, some of the weights that would be included in the rank corresponding to the sparseness ratio when ranked from the smallest absolute value of the weights are disabled. For example, to make the sparseness ratio of a hidden layer Lm 90%, the portion of the weights belonging to the bottom 90% of the absolute values of the weights is disabled. In the following description, a recognition model ML in which the sparseness ratio of at least one hidden layer Lm is increased is referred to as a sparsed recognition model ML.
[0025] In step S12, the accuracy of object recognition by the sparsed recognition model ML is calculated using test images. Different images are used for the test images than the training images. The test images are images obtained by capturing images under the same driving conditions as the driving conditions in which the weight data WD1 being created will be used. Here, in order to create weight data WD1 to be used under the driving conditions of "daytime", "sunny", "highway", and "not congested", the test images are images obtained by capturing images under the driving conditions of "daytime", "sunny", "highway", and "not congested".
[0026] In step S13, it is determined whether the accuracy of object recognition by the sparsified recognition model ML is above a predetermined threshold value. The threshold value may be the same for each driving environment, or it may be different for each driving environment. For example, if the driving environment is a "highway," the threshold value may be lowered to prioritize high-speed calculation by increasing the sparsity ratio, or if the driving environment is a "general road," the threshold value may be raised to prioritize the accuracy of object recognition by lowering the sparsity ratio.
[0027] If the accuracy of object recognition is determined to be below a certain threshold in step S13, the cosine similarity of the feature vectors for each intermediate layer Lm is calculated in step S14. The comparison is between the feature vector output from the intermediate layer Lm before the change in sparseness ratio and the feature vector output from the intermediate layer Lm after the change in sparseness ratio. The cosine similarity is expressed in the range of -1 to +1. The closer the cosine similarity is to +1, the higher the similarity between the comparison targets, and the closer the cosine similarity is to -1, the lower the similarity between the comparison targets. In step S15, the rank of the cosine similarity of each intermediate layer Lm is determined. In step S16, the sparseness ratio of the intermediate layer Lm with the lowest rank in cosine similarity is lowered by a predetermined amount. In this embodiment, the sparseness ratio of the intermediate layer Lm with the lowest rank in cosine similarity, in other words, the intermediate layer Lm with the smallest cosine similarity, is lowered. However, the sparseness ratio of multiple intermediate layers Lm belonging to the lower ranks of cosine similarity may also be lowered. Then, the process returns to step S12, where the object recognition accuracy is calculated again, and in step S13, it is determined whether the object recognition accuracy is equal to or greater than the reference value. The above procedure is repeated until it is determined in step S13 that the object recognition accuracy is equal to or greater than the reference value.
[0028] If it is determined in step S13 that the object recognition accuracy is equal to or greater than the threshold value, the current weight data WD1 is saved in step S17, and the creation of the weight data WD1 is terminated. In the example in Figure 6, after the sparseness ratio of the third hidden layer Lm3, which has the smallest cosine similarity, is changed from 90% to 80% in step S16, the process returns to step S12, and the object recognition accuracy is calculated with the sparseness ratio of the third hidden layer Lm3 being 80%, and the sparseness ratios of the other hidden layers Lm1, Lm2, Lm4, and Lm5 being 90%. However, since the object recognition accuracy is still below the threshold, the process returns to step S12 after the sparseness ratio of the fifth intermediate layer Lm5 is changed from 90% to 80% in step S16, and then the process returns to step S12. The object recognition accuracy is then calculated with the sparseness ratios of the third and fifth intermediate layers Lm3 and Lm5 being 80%, and the sparseness ratios of the other intermediate layers Lm1, Lm2, and Lm4 being 90%. The object recognition accuracy becomes above the threshold when the sparseness ratios of each intermediate layer Lm1 to Lm5 are 80%, 50%, 10%, 10%, and 10%, respectively, and the weight data WD1 at this point is saved.
[0029] Figure 7 is a flowchart showing the procedure for the object recognition method. In this embodiment, the object recognition method is repeatedly performed at a predetermined interval by the object recognition device 100 while the vehicle 10 is running. The object recognition device 100 performs the object recognition method, for example, at a 0.1-second interval.
[0030] In step S21, the object recognition device 100 acquires an image from the camera 20 mounted on the vehicle 10 via the input / output interface 105. In this embodiment, the image acquired by the object recognition device 100 is processed by the image processing unit 110 and then transmitted to the recognition unit 150.
[0031] In step S22, the object recognition device 100 acquires information about the driving environment of the vehicle 10 from various devices 30 to 80 mounted on the vehicle 10 via the input / output interface 105.
[0032] In step S23, the driving environment determination unit 120 determines the driving environment of the vehicle 10 using information acquired from various devices 30 to 80. In this embodiment, the day / night determination unit 121 determines whether the driving environment of the vehicle 10 is daytime or nighttime. The weather determination unit 122 determines whether the driving environment of the vehicle 10 is sunny or rainy. The road determination unit 123 determines whether the driving environment of the vehicle 10 is an expressway or a regular road. The congestion determination unit determines whether the driving environment of the vehicle 10 is congested or not.
[0033] In step S24, the determination unit 130 determines the weight data WD used for object recognition by the recognition model ML, according to the result of the driving environment determination by the driving environment determination unit 120.
[0034] In step S25, the switching unit 140 obtains the weight data WD determined by the determination unit 130 from among the multiple weight data WD1 to WD16 stored in the memory 104, and switches the weight data WD used for object recognition by the recognition model ML. In other words, the switching unit 140 switches the sparseness ratio for each intermediate layer Lm of the recognition model ML according to the judgment result of the driving environment. However, if the weight data WD determined by the determination unit 130 and the weight data WD used for object recognition by the recognition model ML are the same, the switching unit 140 does not need to switch the weight data WD.
[0035] In step S26, the recognition unit 150 uses the recognition model ML and weight data WD to recognize objects in the image acquired from the camera 20 and outputs the object recognition result. After that, the object recognition device 100 terminates the object recognition method. The recognition result output from the recognition unit 150 can be used, for example, to control the vehicle 10 by the vehicle control device 80.
[0036] Figure 8 schematically illustrates how the weight data WD used for object recognition by the recognition model ML is switched according to the driving environment. For example, when the driving environment of vehicle 10 changes from a "highway" to a "general road," the weight data WD used for object recognition by the recognition model ML is switched. As a result, the sparseness ratio of each intermediate layer Lm is switched according to the driving environment.
[0037] As described above, the object recognition device 100 of this embodiment performs object recognition using a sparsified recognition model ML, thus shortening the time required for object recognition by the recognition model ML. In addition, in this embodiment, the sparsity ratio of each intermediate layer Lm of the recognition model ML is dynamically changed according to the driving environment of the vehicle 10, so that a decrease in the accuracy of object recognition by the recognition model ML due to changes in the driving environment can be suppressed. Therefore, the time required for object recognition can be shortened while maintaining the accuracy of object recognition by the recognition model ML.
[0038] Furthermore, in this embodiment, the object recognition device 100 changes the sparseness ratio of the intermediate layer Lm by disabling some of the weights used in the calculations in the intermediate layer Lm using pre-created weight data WD. The accuracy of object recognition using the recognition model ML and weight data WD has been confirmed using test images prepared for each driving environment. Therefore, it is possible to suppress overfitting of the recognition model ML and ensure the accuracy of object recognition by the recognition model ML.
[0039] Furthermore, in this embodiment, the weight data WD is configured to invalidate a portion of the weights used in the calculations in the intermediate layer Lm that are ranked in order of increasing absolute value, according to the sparseness ratio. Therefore, weights that contribute little to the recognition result by the recognition model ML can be invalidated. Thus, it is possible to achieve both the time required for object recognition and the recognition accuracy.
[0040] B. Other embodiments: (B1) In the above embodiment, the object recognition device 100 is equipped with an ISP 101. However, the object recognition device 100 does not need to be equipped with an ISP 101. In this case, the object recognition device 100 may be input with an image after image processing.
[0041] (B2) In the above embodiment, the object recognition device 100 is equipped with an NPU 103, and object recognition by the recognition model ML is performed on the NPU 103. Alternatively, the object recognition device 100 may be equipped with a multi-core CPU capable of parallel processing or a GPU (Graphics Processing Unit) instead of the NPU 103. In this case, object recognition by the recognition model ML may be performed on the multi-core CPU or GPU.
[0042] (B3) In the above embodiment, in the weight data creation method, the sparseness ratio is changed sequentially starting from the intermediate layer Lm with the lowest cosine similarity, and the weight data WD is saved when the accuracy of object recognition exceeds a reference value. Alternatively, the sparseness ratio of the intermediate layer Lm may be changed randomly, and the weight data WD is saved when the accuracy of object recognition is maximized.
[0043] (B4) In the above embodiment, the weight data WD is configured to invalidate a portion of the weights used in the calculations in the intermediate layer Lm that are included in the rank according to the sparseness ratio when the weights are ranked in order from the smallest absolute value. In contrast, the weight data WD may be configured to invalidate a portion that is not included in the rank according to the sparseness ratio when the weights are ranked in order from the smallest absolute value.
[0044] This disclosure is not limited to the embodiments described above, and can be implemented in various configurations without departing from its spirit. For example, the technical features in the embodiments can be replaced or combined as appropriate to solve some or all of the above-described problems, or to achieve some or all of the above-described effects. Furthermore, if a technical feature is not described as essential in this specification, it can be deleted as appropriate.
[0045] [Form 1] The object recognition device (100) comprises: an acquisition unit (105) that acquires images from a camera (20) mounted on a vehicle (10); a driving environment determination unit (120) that determines the driving environment of the vehicle; a switching unit (140) that switches the sparseness ratio of each intermediate layer of a neural network (ML) having a plurality of intermediate layers (Lm) according to the determination result of the driving environment; and a recognition unit (150) that recognizes objects in the image using the neural network. [Form 2] In the object recognition device of Embodiment 1, the switching unit may switch the sparseness ratio of the intermediate layer by disabling some of the weights used in the calculations in the intermediate layer. [Form 3] In the object recognition device of form 2, the switching unit may switch the sparseness ratio of the intermediate layer by disabling some of the multiple weights using pre-created weight data. [Form 4] In the object recognition device of Embodiment 3, the weight data may be configured to invalidate a portion of the weights that are included in the ranking according to the sparseness ratio when the weights are ranked from the smallest absolute value among the plurality of weights. [Form 5] The object recognition method acquires an image from a camera (20) mounted on a vehicle (10), determines the driving environment of the vehicle, switches the sparseness ratio of each intermediate layer of a neural network (ML) having multiple intermediate layers (Lm) according to the determination result of the driving environment, and uses the neural network to recognize an object in the image. [Explanation of symbols]
[0046] 10...Vehicle, 20...Camera, 30...Sun radiation sensor, 40...Rain sensor, 50...Vehicle speed sensor, 60...ETC onboard unit, 70...Navigation system, 80...Vehicle control device, 100...Object recognition device, 101...ISP, 102...CPU, 103...NPU, 104...Memory, 105...Input / output interface, 106...Internal bus, 110...Image processing unit, 120...Driving environment determination unit, 121...Day / night determination unit, 122...Sunny / rainy determination unit, 123...Road determination unit, 124...Traffic congestion determination unit, 130...Decision unit, 140...Switching unit, 150...Recognition unit, ML...Recognition model, WD...Weight data
Claims
1. Object recognition device (100), An acquisition unit (105) that acquires images from a camera (20) mounted on the vehicle (10), A driving environment determination unit (120) that determines the driving environment of the vehicle, A switching unit (140) switches the sparseness ratio for each intermediate layer of a neural network (ML) having multiple intermediate layers (Lm) according to the result of determining the driving environment, A recognition unit (150) that recognizes objects in the image using the neural network, An object recognition device equipped with the following features.
2. The object recognition device according to claim 1, The switching unit switches the sparseness ratio of the intermediate layer by disabling some of the weights used in the calculations in the intermediate layer, an object recognition device.
3. The object recognition device according to claim 2, The switching unit switches the sparseness ratio of the intermediate layer by disabling some of the multiple weights using pre-created weight data, in an object recognition device.
4. The object recognition device according to claim 3, An object recognition device in which the weight data is configured such that, when the weights are ranked from the smallest absolute value among the plurality of weights, a portion of the weight data included in the ranking according to the sparseness ratio is invalidated.
5. A method for recognizing objects, Images are acquired from a camera (20) mounted on the vehicle (10). The driving environment of the aforementioned vehicle is determined, Depending on the result of determining the driving environment, the sparseness ratio of each intermediate layer (Lm) of a neural network (ML) having multiple intermediate layers (Lm) is switched. Using the aforementioned neural network, objects in the image are recognized. Object recognition method.
Citation Information
Patent Citations
Joint sparse method based on mixed particle size used for neural network
JP2022101461A
