Arithmetic apparatus, updating method, and environment recognition apparatus
The computing device updates neural network weights using spatial importance coefficients to reduce computational load and maintain accuracy in parallax estimation, addressing inefficiencies in conventional methods by selectively reducing weights in unnecessary areas.
Patent Information
- Application Number
- JP2023084726
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-23
- Publication Date
- 2024-12-05
AI Technical Summary
Existing machine learning models for parallax calculation in stereo cameras require significant computational resources, and conventional methods of reducing these models lead to accuracy deterioration near spatial information reduction points, causing inference errors.
A computing device that updates neural network weights based on spatial information using importance coefficients, reducing the model by setting weights to zero for areas where estimation is not needed, and performing relearning or fine-tuning to maintain accuracy in relevant regions.
The solution efficiently reduces computational load while maintaining or improving inference accuracy by selectively updating weights based on spatial importance, minimizing errors and resource usage.
Smart Images

Figure 2024168233000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a computing device, an update method, and an environment recognition device. [Background technology]
[0002] Machine learning models are used to calculate disparity information using images captured by a stereo camera. In order to increase the number of pixels to be used for calculating disparity and improve the processing frequency, it is necessary to reduce the amount of calculation. Possible methods for reducing the amount of calculation are to contract the machine learning model and create a lightweight model specialized for the application. In contracting the machine learning model, the amount of calculation is reduced by reducing the redundancy of parameters and approximating the calculation. In creating a lightweight model, the amount of calculation is reduced by creating a model that infers only for spaces that require information in the application and does not infer for spaces that do not require information. For example, in an application for automatic braking, if the braking distance is 25 m, there is little need to recognize objects in spaces closer than 25 m. Therefore, it is possible to reduce the amount of calculation by reducing the disparity search range.
[0003] In reducing the computational load of a disparity model, it is possible to reduce the computational load by first creating a lightweight model specific to the application and then contracting the AI model. In this case, if a model with a relatively large space is created and the spatial information is reduced after applying the contraction technology of the conventional method, the weights are reduced so as to retain the features for the wider space on average during contraction, resulting in the creation of a model that retains the features of the space to be reduced in excess. On the other hand, if the conventional technology is applied after creating a model with reduced spatial information, all inference results are inferred to fall within the range where the spatial information has been reduced, and the accuracy deteriorates in the vicinity of the reduced space. If the conventional technology is applied in this state, it will be affected by the points of accuracy deterioration near the reduced space, and inference errors will occur in the entire inference result.
[0004] Patent Document 1 discloses a calculation device having a neural network that performs calculations using input data and weighting coefficients, characterized in that the calculation device has a network analysis unit that calculates the firing state of neurons in the neural network based on the input data, and a contraction unit that narrows down contraction pattern candidates from a plurality of contraction patterns having set contraction rates for the neural network based on the firing state of the neurons, and performs contraction of the neural network based on the narrowed down contraction pattern candidates to generate a contracted neural network. [Prior art documents] [Patent documents]
[0005] [Patent Document 1] JP 2020-123269 A Summary of the Invention [Problem to be solved by the invention]
[0006] In the invention described in Patent Document 1, there is room for improvement in updating the weights of the neural network model. [Means for solving the problem]
[0007] A computing device according to a first aspect of the present invention comprises an inference unit that outputs an inference result corresponding to input data using an inference model that is a neural network model, and an update unit that updates weights of the inference model based on the inference result and spatial information indicating at least a portion of an area in the input data. An updating method according to a second aspect of the present invention is an updating method executed by a computing device having a memory unit in which an inference model which is a neural network model is stored, and includes an inference process that outputs an inference result corresponding to input data using the inference model, and an updating process that updates weights of the inference model based on the inference result and spatial information indicating at least a portion of an area in the input data. An environmental recognition device according to a third aspect of the present invention is an environmental recognition device that recognizes an external environment using external information acquired by an external sensor, and includes an external environment inference unit that performs inference using a first neural network model to which the external environment information is input, and inference using a second neural network model to which the external environment information is input, and an environmental recognition unit that recognizes the external environment based on the inference result of the external environment inference unit, wherein the first neural network model is a neural network model in which a weight is set by the aforementioned arithmetic device according to spatial information indicating at least a first region in the external environment information, and the second neural network model is a neural network model in which a weight is set by the aforementioned arithmetic device according to spatial information indicating a second region in the external environment information that is different from the first region. Effect of the Invention
[0008] According to the present invention, the weights of the neural network model can be updated in accordance with the set spatial information. [Brief description of the drawings]
[0009] [Figure 1] 1 is a block diagram of a computing device according to a first embodiment. [Diagram 2] FIG. 1 is a diagram showing an example of a coefficient table; [Diagram 3] 1 is a flowchart showing the processing of a computing device according to a first embodiment. [Figure 4] FIG. 13 is a diagram showing an example of a screen output by a setting unit to an input / output device; [Diagram 5] FIG. 11 is a diagram showing a coefficient table in the first modification example. [Figure 6] FIG. 11 is a diagram showing a coefficient table in Modification 2. [Figure 7] 11 is a flowchart showing the process of the arithmetic unit according to the second embodiment. [Figure 8] A flowchart showing details of step S350 in FIG. [Figure 9] FIG. 13 is a block diagram of a computing device according to a third embodiment. [Figure 10]11 is a flowchart showing the process of the arithmetic unit according to the third embodiment. [Figure 11] A configuration diagram of an environment recognition device according to a fourth embodiment. [Figure 12] 13 is a flowchart showing the process of the environment recognition device according to the fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] --First embodiment-- Hereinafter, a first embodiment of a computing device will be described with reference to FIGS.
[0011] 1 is a configuration diagram of a calculation device 1. The calculation device 1 includes a CPU 2 which is a central processing unit, a ROM 3 which is a read-only storage device, a RAM 4 which is a readable and writable storage device, an input / output device 5 which is a user interface, and a communication device 6. The CPU 2 deploys a program stored in the ROM 3 into the RAM 4 and executes it to realize the following functions. That is, an inference unit 41, a loss calculation unit 42, a weight analysis unit 43, a reduction unit 44, and a setting unit 45 are realized by the CPU 2, the ROM 3, and the RAM 4. Hereinafter, the reduction unit 44 will also be referred to as an "update unit."
[0012] The arithmetic device 1 may be realized by a field programmable gate array (FPGA) that is a rewritable logic circuit or an application specific integrated circuit (ASIC) that is an integrated circuit for a specific application, instead of the combination of the CPU 2, the ROM 3, and the RAM 4. The arithmetic device 1 may also be realized by a combination of different configurations, for example, a combination of the CPU 2, the ROM 3, the RAM 4, and an FPGA, instead of the combination of the CPU 2, the ROM 3, and the RAM 4.
[0013] The input / output device 5 is an input / output interface for an operator who uses the arithmetic device 1. The input / output device 5 is, for example, a mouse, a keyboard, and a liquid crystal display. The communication device 6 is an interface, for example, a communication module, for the arithmetic device 1 to communicate with other devices. The arithmetic device 1 exchanges data with a storage device 7. The storage device 7 stores an input image 71, a correct disparity 72, a machine learning model 73, a coefficient table 74, and an estimated disparity 75.
[0014] The input image 71 includes one or more pairs of images. The two images constituting one set have overlapping fields of view, and are, for example, images taken by two cameras arranged with parallel optical axes at a predetermined baseline distance apart. The correct disparity 72 is a set of accurate disparity values (hereinafter also referred to as "correct values") of each pixel in an area where the fields of view of the two images in one set of the input image 71 overlap. For example, in a certain set of images included in the input image 71, the area where the fields of view overlap is 100 pixels wide and 100 pixels high, that is, 10,000 pixels. In this case, the correct disparity 72 includes the correct disparity values corresponding to this set of images, for example, the correct disparity values of each of the 10,000 pixels. The input image 71 and the correct disparity 72 are created in advance and stored in the storage device 7.
[0015] The machine learning model 73 is a model that inputs a pair of images and outputs an estimated disparity 75, which is the disparity of each pixel in an overlapping area in the two images. In the following, the machine learning model 73 may also be called a "neural network model" or an "inference model." In this embodiment, the machine learning model 73 is contracted. Ideally, the estimated disparity 75 coincides with the correct disparity 72. In the following, each value included in the estimated disparity 75 is also called an "estimated value." The coefficient table 74 is a correspondence table between disparities and importance coefficients, which will be described later.
[0016] 2 is a diagram showing an example of the coefficient table 74. The coefficient table 74 is a correspondence table between disparities and importance coefficients. The importance coefficient in this embodiment indicates the degree to which the estimation error of the corresponding disparity is taken into consideration, and takes one of two values, "0" or "1." An importance coefficient of "0" indicates that the estimation error of the corresponding disparity is not taken into consideration in the calculation of a weighting error, which will be described later, and an importance coefficient of "1" indicates that the estimation error of the corresponding disparity is taken into consideration in the weighting error.
[0017] In the example shown in Fig. 2, the importance coefficient is set to "1" when the disparity is 0 to 48 pixels, and the importance coefficient is set to "0" when the disparity is 49 pixels or more. That is, in the example shown in Fig. 2, this means that only pixels with a disparity of 48 pixels or less are taken into account in the weighting error.
[0018] 3 is a flowchart showing the processing executed by the calculation device 1. In this flowchart, steps S301 and S302 are executed by the inference unit 41, steps S303 and S304 are executed by the loss calculation unit 42, steps S305 and S306 are executed by the weight analysis unit 43, and steps S307 to S309 are executed by the contraction unit 44.
[0019] First, in step S301, the inference unit 41 reads the machine learning model 73 and the input image 71. In the following step S302, the inference unit 41 inputs the input image 71 to the machine learning model 73 and executes inference. This inference calculates the estimated disparity 75, for example, an estimated disparity value for each of 10,000 pixels. In the following step S303, the loss calculation unit 42 calculates a disparity error that is the error between the correct value and the inferred value. As described above, the correct value is stored as the correct disparity 72. In the following step S304, the loss calculation unit 42 calculates a weighted error WE that is the product of the importance coefficient corresponding to the correct disparity 72 and the disparity error, using the following Equation 1.
[0020]
number
[0021] In formula 1, dtrue indicates the correct answer, and d est indicates an estimated value. The function f is a function indicating the coefficient table 74, and returns the importance coefficient with the disparity as an argument. In the example shown in FIG. 2, the importance coefficient is set to "1" when the disparity is 0 to 48 pixels, and the importance coefficient is set to "0" when the disparity is 49 pixels or more. true If f(d true ) returns "1", and d true If f(d true ) returns "0." For example, Equation 1 calculates the difference between the inferred values of 10,000 pixels calculated in step S302 and the correct answer value, and then adds up the differences only when the correct answer value is 48 pixels or less to calculate the weighted error WE.
[0022] In step S305, the weight analysis unit 43 calculates a gradient by backpropagation using the weighted error value. In the following step S306, the weight analysis unit 43 calculates the importance using the gradient calculated in step S305 and the weight parameter. For example, the importance X can be calculated as shown in the following formula 2, where the gradient is g and the weight parameter is w.
[0023]
number
[0024] However, the method of calculating the importance is not limited to the above-mentioned method, and any method of calculating the importance using the error between the correct value and the inferred value may be used. In the next step S307, the contraction unit 44 uses the importance values calculated in step S306 to set the weight parameters to 0 or to be contracted in ascending order of importance value so as to achieve the target amount of calculation.
[0025] In the next step S308, the reduction unit 44 performs re-learning or fine tuning using the reduced model. In the final step S309, the reduction unit 44 stores the reduced model and ends the process shown in Fig. 3. Note that the process from step S307 onwards is an example, and various known methods of reduction using the importance of each weight parameter can be used.
[0026] FIG. 4 is a diagram showing an example of a screen 80 that the setting unit 45 outputs to the input / output device 5. The screen 80 has an inference result confirmation area 81 and an importance coefficient setting area 85. The inference result confirmation area 81 includes an input image display button 82, an original inference result display button 83, and a reduced inference result display button 84. When the user presses the input image display button 82, the input / output device 5 displays the input image 71 to be input to the inference unit 41. When the user presses the original inference result display button 83, the input / output device 5 displays the inference result using the machine learning model 73 before reduction. When the user presses the reduced inference result display button 84, the input / output device 5 displays the inference result using the reduced machine learning model 73.
[0027] The importance coefficient setting area 85 includes a disparity input field 86, an importance coefficient input field 87, and a set button 88. When the user inputs numerical values in the disparity input field 86 and the importance coefficient input field 87 and then presses the set button 88, the input contents are added to the coefficient table 74. However, multiple numerical values may be input into the disparity input field 86. The importance coefficient setting area 85 may also include a button for initializing the coefficient table 74.
[0028] According to the above-described first embodiment, the following advantageous effects can be obtained. (1) The arithmetic device 1 includes an inference unit 41 that outputs an estimated disparity 75 that is an inference result corresponding to an input image 71 using a machine learning model 73 that is a neural network model, and a reduction unit 44 that updates the weight of the machine learning model 73 based on the estimated disparity 75 and an importance coefficient that is spatial information indicating at least a portion of an area in the input image 71. Therefore, the arithmetic device 1 can update the weight of the machine learning model 73 in accordance with the set coefficient table 74.
[0029] (2) The contraction unit 44 updates any of the weights of the machine learning model 73 to zero. As a result, the calculation device 1 can contract the machine learning model 73.
[0030] (3) The input images 71 are a pair of images. The estimated disparity 75 is the disparity in an overlapping region of the pair of images that are the input images 71. The coefficient table 74 is a correspondence table of disparity and importance coefficients as shown in FIG. 2. The arithmetic device 1 includes a loss calculation unit 42 that calculates a weighted error based on an error value that is the difference between an inference result and a correct answer value, and the importance coefficient, and a reduction unit 44 that updates the weights of the machine learning model 73 using the weighted error.
[0031] (4) The loss calculation unit 42 calculates a weighted error by adding up the products of the error values and the importance coefficients. As shown in Fig. 2, the importance coefficients include zero. Therefore, the machine learning model 73 can be efficiently contracted by setting the importance coefficients of disparities that do not need to be estimated by the machine learning model 73 to zero.
[0032] (Variation 1) FIG. 5 is a diagram showing a coefficient table 74A in Modification 1. In the above-mentioned first embodiment, the importance coefficient in the coefficient table 74 is either "0" or "1". However, the importance coefficient may have a value other than "0" and "1", and the type of value is not limited to "2". That is, as shown in FIG. 5, it may be an integer from "0" to "4", or a decimal such as "0.1" or "5.5". Even if the value of the importance coefficient or the number of types changes, the process of the flowchart shown in FIG. 3 does not need to be particularly changed. According to Modification 1, the importance of each disparity can be set in detail.
[0033] (Variation 2) 6 is a diagram showing a coefficient table 74B in Modification 2. In the above-mentioned first embodiment and Modification 1, the importance coefficient is set as a value corresponding to the parallax. However, the importance coefficient may be set as a value corresponding to the coordinate space of the input image 71. In this case, the importance coefficient may be a binary value of "0" or "1" as in the first embodiment, or may be three or more types of numbers equal to or greater than "0" as in Modification 1.
[0034] In this modification, on the premise that the area where the fields of view overlap in a pair of images input to the inference unit 41 is known, the importance coefficient of each pixel in the area where the fields of view overlap is defined. For example, when the overlap of the fields of view is 1000 pixels vertically and 1000 pixels horizontally, the upper left coordinate is set to (0, 0) and the lower right coordinate is set to (999, 999) as shown in the lower part of FIG. 6, and the importance coefficient is set for each coordinate space. Specifically, in the example shown in FIG. 6, the area from (0, 0) to (12, 12) in the upper left is set to an importance coefficient of "4". In this modification, the process of the flowchart shown in FIG. 3 does not need to be particularly changed.
[0035] According to this modification 2, the importance coefficient can be set for the coordinate space of the input image 71. Therefore, it is sufficient to set a large importance coefficient for the coordinate space in which the accuracy of parallax is required in the input image 71, and the importance coefficient can be easily set.
[0036] According to the second modification, the following effects can be obtained. (5) The coefficient table 74A is a correspondence table between the coordinates in the overlapping region and the importance coefficients. Therefore, by reducing the importance of regions in the input image 71 that do not require estimation by the machine learning model 73, the machine learning model 73 can be efficiently contracted.
[0037] --Second embodiment-- A second embodiment of the arithmetic device will be described with reference to Figs. 7 and 8. In the following description, the same components as in the first embodiment are given the same reference numerals, and differences will be mainly described. Points that are not particularly described are the same as in the first embodiment. In this embodiment, the main difference from the first embodiment is the content of the processing.
[0038] Fig. 7 is a flowchart showing the processing of the calculation device 1A in the second embodiment. In Fig. 7, the same processing as in Fig. 3 is given the same step number, and similar processing to Fig. 3 is given a similar step number. Steps S301 to S302 are executed by the inference unit 41, steps S303 and S350 are executed by the loss calculation unit 42, steps S305A, S306A, and S360 are executed by the weight analysis unit 43, and steps S307A to S309 are executed by the contraction unit 44.
[0039] Steps S301 to S303 are the same as those in FIG. 3, and therefore their explanation will be omitted. In step S350 executed after step S303, an identical coefficient error calculation process is performed. In the identical coefficient error calculation process, an identical coefficient error, which is an error for each value of the importance coefficient, is calculated. Details of step S350 will be described with reference to FIG. 8.
[0040] Fig. 8 is a flowchart showing the details of step S350 in Fig. 7. First, in step S351, the loss calculation unit 42 identifies the number of types of importance coefficient values in the coefficient table 74. For example, in the example shown in Fig. 2, the importance coefficient can only take the values "0" or "1", so the number of types is "2". In the following step S352, the loss calculation unit 42 selects one of the importance coefficient values that has not been selected. Note that the value selected here will be referred to as the "target coefficient" hereinafter.
[0041] In step S353, the loss calculation unit 42 lists the target disparity, which is the disparity corresponding to the target coefficient. For example, in the example of FIG. 2, when the target coefficient is "1", "0, 1, 2, . . . , 48" are listed as the target disparity in step S353. In the following step S354, the loss calculation unit 42 extracts, from the correct disparity 72, all coordinates of pixels where the correct disparity 72 is the disparity of the target coefficient. Hereinafter, the coordinates extracted in this step are referred to as "extracted coordinates". In the above example, the coordinates where the correct value is "0, 1, 2, . . . , 48" are extracted. In the following step S355, the loss calculation unit 42 extracts the values of the extracted coordinates in the error between the correct value and the inferred value calculated in step S303, calculates the average, and sets it as the identical coefficient error. The identical coefficient error SE V The calculation is expressed by the following Equation 3.
[0042]
number
[0043] In Equation 3, Nv is the total number of extracted coordinates extracted in step S354 when the target coefficient is "V". As shown in Equation 1, this equation is significantly different from the process in step S304 in FIG. 3 in that it does not include the value of the importance coefficient. In the following step S356, the loss calculation unit 42 judges whether or not there is an unselected value in the importance coefficient. If the loss calculation unit 42 judges that there is an unselected value, the process returns to step S352. If the loss calculation unit 42 judges that all values have been selected, in other words, that the processes of steps S352 to S355 have been performed on all values, the process shown in FIG. 8 is terminated and the process returns to FIG. 7. Returning to FIG. 7, the explanation will be continued.
[0044] In step S305A, the weight analysis unit 43 calculates identical coefficient gradients by backpropagation using the identical coefficient errors calculated for each type of importance coefficient calculated in step S350. Comparing the process of this step with step S305 in Fig. 3, the difference is whether the weighted error or the identical coefficient error is used. In the following step S306A, the weight analysis unit 43 calculates the importance for each gradient calculated for each type of importance coefficient in the same manner as step S306 in Fig. 3. Note that the step numbers are different from those in Fig. 3 for ease of explanation.
[0045] In the next step S360, the weight analysis unit 43 calculates the sum of the products of the importance (same coefficient importance) calculated for each importance coefficient calculated in step S306A and the importance coefficient as the weighted importance. The weighted importance is a matrix of the same dimension as the weight W. In the next step S307A, the contraction unit 44 searches for a weight parameter to be contracted using the weighted importance calculated in step S360 instead of the importance in step S307. The processes in the next steps S308 and S309 are the same as those in FIG. 3, and therefore will not be described.
[0046] According to the above-described second embodiment, the following advantageous effects can be obtained. (6) The coefficient table 74 is a correspondence table between disparity and importance coefficient. The calculation device 1A includes a loss calculation unit 42 that calculates an error based on an error value that is the difference between an inference result and a correct answer value, a weight analysis unit 43 that calculates the importance of a weight parameter in a neural network model using the error (S306A in FIG. 7) and calculates a weighting importance based on the importance of the weight parameter and the importance coefficient (S360), and a contraction unit 44 that determines a weight of an update candidate based on the weighting importance. Therefore, the weighting importance is calculated using the coefficient table 74, and the machine learning model 73 can be contracted. Note that there is no clear superiority or inferiority between the method of the first embodiment and the method of the present embodiment, and the same effect can be obtained with either method.
[0047] --Third embodiment-- A third embodiment of the arithmetic device will be described with reference to Figs. 9 and 10. In the following description, the same components as in the first embodiment are given the same reference numerals, and differences will be mainly described. Points that are not particularly described are the same as in the first embodiment. This embodiment differs from the first embodiment mainly in that learning is performed instead of reduction.
[0048] 9 is a configuration diagram of a calculation device 1B. The difference from the calculation device 1 in the first embodiment is that the RAM 4 includes a learning unit 44A instead of the contraction unit 44. The learning unit 44A updates the weight of the machine learning model 73 so that the inference result by the inference unit 41 using the machine learning model 73 becomes closer to the true value. Hereinafter, the learning unit 44A is also referred to as an "update unit."
[0049] Fig. 10 is a flowchart showing the processing of the arithmetic device 1B in the third embodiment. In Fig. 10, the same processing as in Fig. 3 is given the same step number, and similar processing to Fig. 3 is given a similar step number. Steps S301 to S302 are executed by the inference unit 41, steps S303 and S304 are executed by the loss calculation unit 42, step S305 is executed by the weight analysis unit 43, and steps S380 and S309 are executed by the learning unit 44A. Compared with Fig. 3 in the first embodiment, it is different in that step S306 for calculating the importance is deleted, and step S380 is added instead of steps S307 and S308.
[0050] The processing in steps S301 to S305 is the same as in the first embodiment, and therefore the description will be omitted. In step S380, the learning unit 44A changes the weight parameters of the machine learning model 73 by the gradient descent method. In the following step S309A, the learning unit 44A saves the machine learning model 73 changed in step S380, and ends the processing shown in FIG.
[0051] According to the above-described third embodiment, the following advantageous effects can be obtained. (7) The learning unit 44A updates the weights of the machine learning model 73 so that the inference result becomes closer to the true value.
[0052] --Fourth embodiment-- An environment recognition device will be described as a fourth embodiment of the present invention with reference to Fig. 11 and Fig. 12. In the following description, the same components as those in the first embodiment are given the same reference numerals, and differences will be mainly described. This embodiment differs from the first embodiment in that the environment recognition device is equipped with an updated or contracted machine learning model 73 generated by a calculation device 1.
[0053] 11 is a configuration diagram of the environment recognition device 300. The environment recognition device 300 is mounted on a vehicle together with a first imaging device 310, a second imaging device 311, and a vehicle control device 340, and recognizes the external environment. However, the object on which the environment recognition device 300 is mounted is not limited to a vehicle, and may be any movable body. In this case, the first imaging device 310, the second imaging device 311, and the vehicle control device 340 are also mounted on the same movable body.
[0054] The environment recognition device 300 includes an image acquisition unit 320 that acquires images captured by the first imaging device 310 and the second imaging device 311, and a disparity calculation unit 330 that performs disparity estimation. Note that hereinafter, the feature extraction unit 331, the disparity inference unit 333, and the mask inference unit 334 are collectively referred to as an "external world inference unit," and the mask disparity image generation unit 335 is also referred to as an "environment recognition unit."
[0055] The disparity calculation unit 330 includes a feature extraction unit 331 that extracts features from an acquired image, a machine learning model storage unit 332 that stores a machine learning model (called a "neural network model" or an "inference model"), a disparity inference unit 333 that determines the disparity, a mask inference unit 334 that estimates a mask region described later, and a mask disparity image generation unit 335 that generates a disparity image based on the disparity determined by the disparity inference unit 333 and the mask region determined by the mask inference unit 334. The image feature extraction in the feature extraction unit 331 and the inference processing in the disparity inference unit 333 and the mask inference unit 334 are processed using a machine learning model. The mask disparity image generated by the mask inference unit 334 is output to a downstream vehicle control device 340 and is used for executing an arbitrary vehicle control application, etc.
[0056] The first imaging device 310 and the second imaging device 311 are, for example, stereo cameras that capture images of the area in front of the vehicle to generate a pair of captured images (a left image P1 and a right image P2) having a mutual parallax. The first imaging device 310 and the second imaging device 311 each have an image sensor built in and collect information about the outside world of the vehicle. Therefore, the first imaging device 310 and the second imaging device 311 can also be called "external world sensors."
[0057] This stereo camera has a first imaging device 310 as a left camera and a second imaging device 311 as a right camera. The left camera and the right camera are arranged, for example, near the upper part of the windshield of the vehicle, spaced apart by a predetermined distance along the width direction of the vehicle. In addition, the left camera and the right camera are set to perform imaging operations in synchronization with each other. A stereo parallax image can be generated by utilizing the parallax between the left and right cameras.
[0058] The first imaging device 310 and the second imaging device 311 may generate a pair of captured images having a parallax between them. For example, the first imaging device 310 and the second imaging device 311 may be included in a triple-lens camera having three or more cameras. The first imaging device 310 and the second imaging device 311 may be included in a multi-camera that captures images including an overlapping imaging area around the vehicle. Furthermore, the first imaging device 310 and the second imaging device 311 may be different types of cameras. Also, they may be a single-lens camera instead of multiple cameras. For example, a single-lens camera having a moving mechanism may be treated as the first imaging device 310 when capturing images at a first position, and may be treated as the second imaging device 311 when capturing images at a second position.
[0059] The image acquisition unit 320 acquires a left image P1 acquired by the first imaging device 310 and a right image P2 acquired by the second imaging device 311. At this time, each of the left image P1 and the right image P2 may be a single image or a plurality of images acquired in time series.
[0060] The feature extraction unit 331 extracts features for disparity estimation from the left image P1 and the right image P2 acquired by the image acquisition unit 320. A machine learning model can be used to extract the features. The feature extraction unit 331 inputs the left image P1 and the right image P2 having disparity to a machine learning model for feature estimation, extracts features from the left image P1 and the right image P2, and generates a feature map K corresponding to each of them. The feature map K is composed of pixel values having a width W, a height H, and a channel C arranged three-dimensionally, for example.
[0061] Next, the feature extraction unit 331 creates information about corresponding points in the left image P1 and the right image P2 (hereinafter referred to as "feature U for disparity estimation") using the two generated feature maps K. Here, the information about corresponding points in the left image P1 and the right image P2 is information for calculating the amount of displacement according to the depth (distance) of the same subject in the left image P1 and the right image P2.
[0062] However, the feature extraction unit 331 may extract features from a plurality of time-series left images P1 acquired by the image acquisition unit 320, and create information on corresponding points between time-series feature maps K. Note that a predetermined preprocessing may be performed prior to the extraction of features. Examples of preprocessing include affine processing, luminance correction processing, pixel complementation processing, and the like. By extracting features using preprocessed images, the amount of calculation of the machine learning model for feature estimation can be reduced.
[0063] The disparity inference unit 333 reads out weight parameters of the machine learning model (hereinafter, also referred to as "first neural network model") from the machine learning model storage unit 332. The weight parameters indicate the reduced machine learning model updated by the arithmetic device 1 according to the first embodiment. That is, the updated machine learning model is updated using an importance coefficient for a predetermined disparity. The disparity inference unit 333 controls calculation so as to omit calculations such as multiplication and addition related to the weight parameters whose importance coefficients are set to 0. The machine learning model storage unit 332 stores information on the weight parameters related to the updated machine learning model updated by the arithmetic device 1 according to the first embodiment.
[0064] However, the information on the weight parameters stored in the machine learning model storage unit 332 may be the numerical values of the weight parameters themselves, or may be in an encoded format. The machine learning model storage unit 332 may store weight parameters in advance, or may store weight parameters received from a server (not shown) via a network (not shown). Furthermore, the machine learning model storage unit 332 may function only as a communication interface, and may obtain weight parameters from a server (not shown) whenever a request is received from the feature extraction unit 331 or the like.
[0065] The disparity inference unit 333 estimates disparity using the updated machine learning model related to the weighting parameters read from the machine learning model storage unit 332, using the disparity estimation feature U as an input value. Here, since the updated machine learning model is contracted based on the importance coefficient for a specific disparity, the estimation accuracy for the disparity in a region where the importance coefficient is set high (hereinafter, also referred to as the "first region") is high. For example, when the importance coefficient of the disparity corresponding to a region close to the vehicle is set high and the importance coefficient of the disparity corresponding to a region far from the vehicle is set low, the updated machine learning model has high inference accuracy in a range close to the vehicle by updating using the importance coefficient. Therefore, by using the updated machine learning model, disparity can be inferred with the accuracy required for each vehicle control application. The disparity inference unit 333 outputs a disparity image D indicating the pixel value of each pixel obtained by inference to the mask disparity image generation unit 335 at the subsequent stage.
[0066] On the other hand, in a region where the importance coefficient is set low (hereinafter, also referred to as a "second region"), the parallax inference accuracy may decrease with the execution of the update. Therefore, the environment recognition device 300 according to the fourth embodiment has a mask inference unit 334 that performs a mask (occlusion) process on the region where the parallax inference accuracy is low.
[0067] The mask inference unit 334 reads weight parameters related to the mask estimation machine learning model (hereinafter also referred to as a “second neural network model”) from the machine learning model storage unit 332, and generates a mask based on the disparity estimation feature U and the mask estimation machine learning model. Specifically, the mask inference unit 102 inputs the disparity estimation feature U to the mask estimation machine learning model, and obtains data output from the mask estimation machine learning model as mask region data M.
[0068] The mask region data M is data for shielding or invalidating a region (hereinafter referred to as a "mask region") in which the estimation accuracy of the parallax may be reduced due to contraction based on the importance coefficient among the second regions in which the importance coefficient is set low. For example, the mask region data M is set such that a first value (e.g., "0") is set to pixels corresponding to the inside of the mask region, and a second value (e.g., "1") different from the first value is set to pixels corresponding to the outside of the mask region. Note that the value set to each pixel of the mask region data M does not have to be binary data, and may be a continuous value. The mask inference unit 334 outputs the mask region data M obtained by inference, i.e., the first value or the second value set for each pixel, to the mask parallax image generation unit 335 at the subsequent stage.
[0069] The weighting parameters of the mask estimation machine learning model are trained to set a first value to pixels corresponding to spatial information in which the importance coefficient is set low, and set a second value to pixels corresponding to spatial information in which the importance coefficient is set high. A predetermined threshold value for the importance coefficient, an evaluation function, or the like may be appropriately used as a criterion for setting the first value or the second value.
[0070] The mask parallax image generating unit 335 generates a mask parallax image Dm based on the parallax image D and the mask region data M input from the parallax inference unit 333 and the mask inference unit 334. At this time, the parallax of the pixel to which the value of "0" is set by the mask inference unit 334 is invalidated, and the parallax value obtained by the parallax inference unit 333 is used for the pixel to which the value of "1" is set. The mask region data M generated by the mask inference unit 334 can invalidate the pixel corresponding to the parallax that is expected to reduce the accuracy of the parallax inference, and it is possible to suppress erroneous detection and erroneous recognition. The generated parallax image D is output to the vehicle control device 340 and used for any vehicle control application, self-position estimation, etc.
[0071] Fig. 12 is a flowchart showing the processing of the environment recognition device 300. Specifically, Fig. 4 shows the processing of the disparity inference unit 333, the mask inference unit 334, and the mask disparity image generation unit 335. First, in step S31, the disparity inference unit 333 inputs the disparity estimation feature amount U to the updated machine learning model. In the following step S32, the mask inference unit 334 inputs the disparity estimation feature amount U to the mask estimation machine learning model. In the following step S33, the disparity inference unit 333 executes inference using the updated machine learning model and outputs a disparity image D.
[0072] In the next step S34, the mask inference unit 334 executes inference using the mask estimation machine learning model, and outputs the mask region data M. Finally, in step S35, the mask parallax image generation unit 335 generates a mask parallax image Dm, and ends the process shown in Fig. 12. Note that steps S31 and S32 may be executed in reverse order or in parallel. Also, steps S33 and S34 may be executed in reverse order or in parallel.
[0073] According to the fourth embodiment described above, the following advantageous effects can be obtained. (8) The environment recognition device 300 recognizes the external environment using external information acquired by external sensors built into the first imaging device 310 and the second imaging device 311. The environment recognition device 300 includes an external inference unit that performs inference using a first neural network model to which external information is input, and an inference using a second neural network model to which external information is input, that is, a feature extraction unit 331, a parallax inference unit 333, and a mask inference unit 334, and an environment recognition unit that recognizes the external environment based on the inference result of the external inference unit, that is, a mask parallax image generation unit 335. The first neural network model is a machine learning model after updating that is updated by the arithmetic device 1 according to the first embodiment using an importance coefficient for a predetermined parallax. The second neural network model is a neural network model in which a weight is set by the arithmetic device 1 according to spatial information indicating a second area different from a first area in the external information.
[0074] (9) The external sensor provided in the first imaging device 310 and the second imaging device 311 is an image sensor. The external information is a captured image captured using the external sensor, specifically, a left image P1 and a right image P2. The feature extraction unit 331, the parallax inference unit 333, and the mask inference unit 334 infer the parallax of the captured image using a first neural network model, and infer whether or not to set a mask on a pixel in the captured image using a second neural network model. The mask parallax image generation unit 335 (environment recognition unit) generates a parallax image in which a mask is set on a pixel inferred to be set on the pixel based on the estimation results of the feature extraction unit 331, the parallax inference unit 333, and the mask inference unit 334.
[0075] Therefore, even if an area where the disparity inference accuracy is reduced exists when an updated machine learning model specialized for a specific application is applied to the environment recognition device 300, it is possible to prevent erroneous detection and erroneous recognition. Although an example of an environment recognition device that estimates disparity has been shown in the fourth embodiment, recognition of a task such as depth estimation or estimating the type of an object reflected in an image on a pixel-by-pixel basis may also be used.
[0076] In each of the above-mentioned embodiments and modifications, the configuration of the functional blocks is merely an example. Some functional configurations shown as separate functional blocks may be integrated, or a configuration shown in one functional block diagram may be divided into two or more functions. In addition, some of the functions of each functional block may be provided by other functional blocks.
[0077] In the above-mentioned embodiments and modifications, the program is stored in the ROM 3, but the program may be stored in a non-volatile storage device built into the arithmetic device 1. The arithmetic device 1 may also be provided with an input / output interface (not shown), and the program may be read from another device via the input / output interface and a medium available to the arithmetic device 1 when necessary. The medium here refers to, for example, a storage medium that is detachable from the input / output interface, or a communication medium, i.e., a network such as a wired, wireless, or optical network, or a carrier wave or digital signal that propagates through the network. Also, some or all of the functions realized by the program may be realized by a hardware circuit or an FPGA.
[0078] The above-mentioned embodiments and modifications may be combined with each other. Although various embodiments and modifications have been described above, the present invention is not limited to these. Other aspects that are conceivable within the scope of the technical idea of the present invention are also included in the scope of the present invention. [Explanation of symbols]
[0079] 1, 1A, 1B: arithmetic device, 5: input / output device, 6: communication device, 7: storage device, 41: inference unit, 42: loss calculation unit, 43: weight analysis unit, 44: reduction unit, 44A: learning unit, 45: setting unit, 41: input image, 72: correct disparity, 73: machine learning model, 74, 74A, 74B: coefficient table, 300: environment recognition device, 331: feature extraction unit (external world inference unit), 333: disparity inference unit (external world inference unit), 334: mask inference unit (external world inference unit), 335: mask disparity image generation unit (environment recognition unit)
Claims
1. an inference unit that outputs an inference result corresponding to input data using an inference model that is a neural network model; A computing device comprising: an update unit that updates the weights of the inference model based on the inference result and spatial information indicating at least a portion of the area in the input data.
2. 2. The computing device according to claim 1, The update unit is a calculation device that updates the weights of any of the inference models to zero.
3. 2. The computing device according to claim 1, The update unit is a calculation device that updates the weights of the inference model so that the inference result becomes closer to the true value.
4. 2. The computing device according to claim 1, the input data is a set of images; the inference result is a disparity in an overlapping region of the set of images; the spatial information is a correspondence table of disparity and importance coefficient; a loss calculation unit that calculates a weighted error based on an error value that is a difference between the inference result and a correct answer value and the importance coefficient, The update unit is a computing device that determines the weight of the inference model using the weighting error.
5. 5. The computing device according to claim 4, the loss calculation unit calculates the weighted error by adding up a product of the error value and the importance coefficient; An arithmetic unit in which the importance coefficient includes zero.
6. 2. The computing device according to claim 1, the input data is a set of images; the inference result is a disparity in an overlapping region of the set of images; the spatial information is a correspondence table of coordinates and importance coefficients in the overlapping region; a loss calculation unit that calculates a weighted error based on an error value that is a difference between the inference result and a correct answer value and the importance coefficient, The update unit is a computing device that uses the weighting error to determine the weight of the inference model.
7. 2. The computing device according to claim 1, the input data is a set of images; the inference result is disparity in the set of images; the spatial information is a correspondence table of disparity and importance coefficient; a loss calculation unit that calculates an error based on an error value that is a difference between the inference result and a correct answer; and a weight analysis unit that calculates the importance of a weight parameter in the inference model using the error, and calculates a weighting importance based on the importance of the weight parameter and the importance coefficient. The update unit determines a weight for the update candidate based on the weighted importance.
8. An update method executed by a computing device having a memory unit in which an inference model that is a neural network model is stored, comprising: an inference process that outputs an inference result corresponding to input data using the inference model; An updating method including an updating process that updates weights of the inference model based on the inference result and spatial information indicating at least a portion of the area in the input data.
9. An environment recognition device that recognizes an external environment using external information acquired by an external sensor, an external environment inference unit that performs inference using a first neural network model to which the external environment information is input, and inference using a second neural network model to which the external environment information is input; An environment recognition unit that recognizes an external environment based on an inference result of the external environment inference unit, the first neural network model is a neural network model in which a weight is set according to spatial information indicating at least a first region in the external world information by the arithmetic device according to claim 1; The second neural network model is a neural network model in which weights are set by the calculation device described in claim 1 according to spatial information indicating a second area in the external world information that is different from the first area.
10. The environment recognition device according to claim 9, the external sensor is an image sensor, the external environment information is an image captured by the external environment sensor, the external world inference unit infers a parallax of the captured image using the first neural network model, and infers whether or not to set a mask on a pixel in the captured image using the second neural network model; The environment recognition unit generates a disparity image in which a mask is set for pixels inferred to have a mask set based on an estimation result of the inference unit.
Citation Information
Patent Citations
Arithmetic unit
JP2020123269A
