Inference device, learning device, inference method, inference program, learning method, and learning program

Abstract

The present invention provides an inference device that can suppress the decrease in detection sensitivity due to fluctuations in the detection environment in multimodal sensing. [Solution] An inference device comprising: a first feature extraction unit that extracts a first feature from a first signal; a second feature extraction unit that extracts a second feature from a second signal; a feature fusion unit that weights and fuses the first feature and the second feature, respectively; a specific feature extraction unit that extracts specific features used for object recognition from the fused features; a recognition unit that recognizes the object using the specific features; and a weight adjustment unit that adjusts the weights used by the feature fusion unit to weight the first feature and the second feature, respectively, based on at least one of the recognition accuracy of the object by the recognition unit, the evaluation result of the task based on the recognition result of the object by the recognition unit, or environmental information corresponding to the environment when the first signal and the second signal are detected.

Description

Technical Field

[0001] The present invention relates to an inference device, a learning device, an inference method, an inference program, a learning method, and a learning program.

Background Art

[0002] In recent years, tasks such as object detection have been performed based on multi-modal data detected by a plurality of different modalities.

[0003] In such multi-modal sensing, for example, an image captured by a camera and a distance measurement result by LiDAR (Light Detection And Ranging) are fused and used for task execution.

[0004] The following Patent Document 1 discloses the following prior art. A distance measurement device that measures the distance to an object based on a plurality of signals having different formats, including a signal conversion unit and a distance information output unit. The signal conversion unit matches the format of a second signal related to the distance to the object to the format of a first signal related to an image. The signal conversion unit converts the length information in at least one dimension of the plurality of dimensions of the first signal and the length information in the plurality of dimensions of the second signal whose format is matched to the first signal so that they represent the same length information in the same direction. The distance information output unit outputs distance information based on the first signal and the second signal whose format is matched to the first signal, which has been machine-learned in advance.

[0005] The following Non-Patent Document 1 discloses the following prior art. An image captured by a camera and a point cloud measured by LiDAR are converted into data in a common bird's-eye view, and feature amounts are extracted respectively. Then, the extracted feature amounts are fused using a machine-learned encoder and used for tasks such as object detection.

Prior Art Documents

Patent Documents

[0006] [Patent Document 1] Japanese Patent Publication No. 2021-12133 [Non-patent literature]

[0007] [Non-Patent Document 1] Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang, Huizi Mao, Daniela Rus, and Song Han, BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation, 26 May 2022,<https: / / arxiv.org / abs / 2205.13542> [Overview of the project] [Problems that the invention aims to solve]

[0008] The object detection sensitivity of each modality may vary depending on the detection environment. For example, when detecting an object using distance measurement results from LiDAR, on a sunny day, reflected light from direct sunlight shining on the object may act as a disturbance, potentially reducing the detection sensitivity of that object. Also, when detecting an object using images captured by a camera, at night or in rainy weather, the brightness of the image may decrease, potentially reducing the detection sensitivity of that object. The above prior art uses detection results from multiple modalities, but when fusing the multiple detection results, variations in the detection environment are not taken into consideration, so the accuracy of object detection may decrease due to variations in the detection environment.

[0009] The present invention was made to solve the above-mentioned problems. Specifically, the present invention aims to provide an inference device, a learning device, an inference method, an inference program, a learning method, and a learning program that can suppress the decrease in detection sensitivity due to fluctuations in the detection environment in multimodal sensing. [Means for solving the problem]

[0010] The above-mentioned problems of the present invention are solved by the following means.

[0011] (1) An inference device comprising: a first feature extraction unit for extracting a first feature from a first signal; a second feature extraction unit for extracting a second feature from a second signal; a feature fusion unit for weighting and fusing the first feature and the second feature, respectively; a specific feature extraction unit for extracting specific features used for object recognition from the fusing features; a recognition unit for recognizing the object using the specific features; and a weight adjustment unit for adjusting the weights used by the feature fusion unit to weight the first feature and the second feature, respectively, based on the recognition accuracy of the object by the recognition unit, the evaluation result of the task based on the recognition result of the object by the recognition unit, or environmental information corresponding to the environment when the first signal and the second signal are detected.

[0012] (2) The inference apparatus described in (1) above, wherein the first signal is two-dimensional image data and the second signal is three-dimensional point cloud data obtained from LiDAR.

[0013] (3) The recognition unit is the inference device described in (1) above, which recognizes the position of the object using the specific feature quantity.

[0014] (4) The inference apparatus according to (1) above, wherein the recognition unit outputs the recognition result of the object along with the recognition accuracy of the object, and the weight adjustment unit determines the weights for the first feature and the second feature, respectively, based on the recognition accuracy of the object output by the recognition unit.

[0015] (5) The inference device according to (4) above, wherein the weight adjustment unit has a learning model that outputs weights for the first feature and the second feature, respectively, in response to input of at least the recognition accuracy of the object by the recognition unit and the environment-related information, and the learning model is learned by reinforcement learning which is rewarded by improving the recognition accuracy of the object by the recognition unit.

[0016] (6) The inference apparatus according to (1) above, wherein the weight adjustment unit determines the weights for the first feature and the second feature based on the environment information, using a table that defines the correspondence between the environment information and the weights for the first feature and the second feature, respectively.

[0017] (7) The inference device according to (4) above, wherein the weight adjustment unit has a learning model that outputs weights for the first feature and the second feature, respectively, in response to the input of the environment-responsive information, and the learning model is trained using training data of combinations of the first signal and the second signal and the correct recognition result of the object.

[0018] (8) A learning device having a control unit that trains the learning model described in (5) above by reinforcement learning, with the improvement of the recognition accuracy of the object by the recognition unit as the reward.

[0019] (9) A learning device having a control unit that trains the learning model described in (7) above using training data of combinations of the first signal and the second signal and the correct recognition result of the object.

[0020] (10) An inference method comprising: (a1) extracting a first feature from a first signal; (b1) extracting a second feature from a second signal; (c1) weighting and fusing the first feature and the second feature, respectively; (d1) extracting a specific feature from the fusing feature to be used for object recognition; (e1) recognizing the object using the specific feature; and (f1) adjusting the weights used to weight the first feature and the second feature in step (c1) based on the recognition result of the object recognized in step (e1), the evaluation result of the task based on the recognition result of the object in step (ei), or environmental information corresponding to the environment when the first signal and the second signal are detected.

[0021] (11) A step (a2) of extracting a first feature amount from a first signal, a step (b2) of extracting a second feature amount from a second signal, a step (c2) of weighting and fusing the first feature amount and the second feature amount respectively, a step (d2) of extracting a specific feature amount used for object recognition from the fused feature amount, a step (e2) of recognizing the object using the specific feature amount, and based on the recognition result of the object recognized in step (e2), the evaluation result of a task based on the recognition result of the object in step (e2), or environmental response information corresponding to the environment when detecting the first signal and the second signal, a step (f2) of adjusting the weights for weighting the first feature amount and the second feature amount in step (c2). An inference program for causing a computer to execute a process having these steps.

[0022] (12) A learning method having a step (a3) of learning the learning model described in (5) above by reinforcement learning with the improvement of the recognition accuracy of the object by the recognition unit as a reward.

[0023] (13) A learning program for causing a computer to execute a process having a step (a4) of learning the learning model described in (5) above by reinforcement learning with the improvement of the recognition accuracy of the object by the recognition unit as a reward.

[0024] (14) A learning method having a step (a5) of learning the learning model described in (7) above using teacher data of a combination of the first signal and the second signal and the correct answer of the recognition result of the object.

[0025] (15) A learning program for causing a computer to execute a process having a step (a6) of learning the learning model described in (7) above using teacher data of a combination of the first signal and the second signal and the correct answer of the recognition result of the object.

Advantages of the Invention

[0026] Features are extracted from two different signals, weighted with adjusted weights, and fused to form a fused set of features. A specific feature used for object recognition is then extracted from this fused feature, and the object is recognized using this specific feature. The weights are adjusted based on the object recognition accuracy, the task evaluation results based on the object recognition results, or environmental information corresponding to the environment. This suppresses the decrease in detection sensitivity due to fluctuations in the detection environment in multimodal sensing. [Brief explanation of the drawing]

[0027] The advantages and features provided by one or more embodiments of the present invention will be better understood from the following detailed description and accompanying drawings. However, these are for illustrative purposes only and are not intended to limit the present invention. [Figure 1] This is a diagram showing the schematic configuration of the inference device. [Figure 2] This is a block diagram showing the hardware configuration of the inference device. [Figure 3] This is a flowchart showing the operation of the inference device. [Figure 4] This is an explanatory diagram illustrating the first embodiment of the inference device. [Figure 5] This is an explanatory diagram illustrating a second embodiment of the inference device. [Figure 6] This is a diagram showing the schematic configuration of the inference device. [Figure 7A] This is a diagram showing image data captured by a camera at night. [Figure 7B] This figure shows an image created from point cloud data detected by LiDAR at night. [Figure 8] This is a diagram showing the schematic configuration of the inference device. [Figure 9] This is a flowchart showing the operation of the inference device during inference. [Figure 10] This flowchart shows the operation of the weight adjustment unit of the inference device during the training of the learning model. [Figure 11] This is a diagram showing the schematic configuration of the inference device. [Figure 12]This is a flowchart showing the operation of the inference device. [Modes for carrying out the invention]

[0028] Hereinafter, an inference device, a learning device, an inference method, an inference program, a learning method, and a learning program according to embodiments of the present invention will be described with reference to the attached drawings. However, the scope of the present invention is not limited to the disclosed embodiments. In the description of the drawings, the same elements are denoted by the same reference numerals, and redundant descriptions are omitted. Also, the dimensional ratios in the drawings are exaggerated for illustrative purposes and may differ from the actual ratios.

[0029] (First Embodiment) Figure 1 is a diagram showing the schematic configuration of inference device 1. Figure 2 is a block diagram showing the hardware configuration of inference device 1.

[0030] As shown in Figure 1, the inference device 1 includes a camera encoder 110, a LiDAR encoder 120, a feature fusion unit 130, a specific feature extraction unit 140, an object recognition unit 150, and a weight adjustment unit 160. The inference device 1 may further include a camera 600 and a LiDAR 700. The inference device 1 can be constructed by a computer including a control unit 100, a storage unit 200, a display unit 300, an input unit 400, and a communication unit 500, as shown in Figure 2. The functions of the camera encoder 110, LiDAR encoder 120, feature fusion unit 130, specific feature extraction unit 140, object recognition unit 150, and weight adjustment unit 160 are realized by the control unit 100 executing a program.

[0031] As shown in Figure 2, the control unit 100, storage unit 200, display unit 300, input unit 400, and communication unit 500 of the inference device 1 are interconnected via a bus.

[0032] The control unit 100 is composed of a CPU (Central Processing Unit) and performs control and calculation processing of each part of the inference device 1 according to the program. The functions of the control unit 100 will be described later.

[0033] The storage unit 200 may consist of RAM (Random Access Memory), ROM (Read Only Memory), and flash memory. The RAM temporarily stores programs and data as a working area for the control unit 100. The ROM stores various programs and data in advance. The flash memory stores various programs and data, including the operating system.

[0034] The display unit 300 is, for example, a liquid crystal display, which displays various information.

[0035] The input unit 400 is comprised of, for example, a touch panel and various keys. The input unit 400 is used for various operations and inputs.

[0036] The communication unit 500 is an interface for communicating with external devices. Network interfaces conforming to standards such as Ethernet (registered trademark), SATA, PCI Express, USB, and IEEE 1394 may be used for communication. Other local connection interfaces such as Bluetooth (registered trademark) and IEEE 802.11 wireless communication interfaces may also be used.

[0037] Camera 600 outputs the captured image. Camera 600 can output the image as two-dimensional image data. Camera 600 is, for example, a visible light camera. Camera 600 may also be a near-infrared camera. The image output from camera 600 constitutes the first signal. Hereinafter, the image data output from camera 600 will also be simply referred to as "image data".

[0038] The LiDAR700 measures the distance to an object illuminated by a laser pulse by detecting the reflected light using the Time of Flight (TOF) method. The LiDAR700 outputs 3D point cloud data, using the measured distance values ​​as pixel values. The point cloud data output from the LiDAR700 constitutes the second signal. Hereafter, the point cloud data output from the LiDAR700 will also be simply referred to as "point cloud data".

[0039] The functions of the control unit 100 will be explained with reference to Figure 1.

[0040] The camera encoder 110 can extract features from image data captured by the camera 600. The camera encoder 110 extracts features, for example, by convolutional operations using a neural network. Hereinafter, the features extracted from image data by the camera encoder 110 will also be referred to as "first features". The camera encoder 110 constitutes the first feature extraction unit.

[0041] The LiDAR encoder 120 can extract features from the point cloud data output from the LiDAR 600. The LiDAR encoder 120 extracts features, for example, by convolutional operations using an encoder. Hereinafter, the features extracted from the point cloud data by the LiDAR encoder 120 will be referred to as "secondary features". The LiDAR encoder 120 constitutes the secondary feature extraction unit.

[0042] The feature fusion unit 130 weights the first and second features using weights adjusted by the weight adjustment unit 160. Specifically, the feature fusion unit 130 weights the first and second features extracted from image data and point cloud data captured and measured at the same time by the camera 600 and LiDAR 700, respectively. In this process, timestamps added to the image and point cloud data may be used. The feature fusion unit 130 then fuses the weighted first and second features.

[0043] More specifically, the feature fusion unit 130 matches the coordinates of the first and second features extracted from the image data and point cloud data, respectively, at the same time point, and performs a process to align their dimensions. This makes the number of dimensions of the first and second features the same. Then, the feature fusion unit 130 weights the first and second features, which have had their dimensions aligned, and fuses them. That is, the feature fusion unit 130 multiplies the scalar amount for each dimension of the first feature by the weight adjusted for the first feature, and multiplies the scalar amount for each dimension of the second feature by the weight adjusted for the second feature. The feature fusion unit 130 can fuse the two by adding the weighted first and second features for each corresponding dimension.

[0044] The specific feature extraction unit 140 extracts specific features from the features merged by the feature fusion unit 130 that the object recognition unit 150 uses to recognize objects. The specific feature extraction unit 140 extracts specific features, for example, by a convolution operation using an encoder.

[0045] The object recognition unit 150 recognizes objects using specific features. Specifically, for example, the object recognition unit 150 recognizes the position of an object using specific features. The object recognition unit 150 may recognize the position of an object as the coordinates of the object on the image data output from the camera 600. The object recognition unit 150 may also recognize the position of an object as the coordinates of the object on the image corresponding to the point cloud data output from the LiDAR 700.

[0046] The object recognition unit 150 may calculate the likelihood of an object being present at a given location, along with the object's position. Hereinafter, the likelihood of an object being present at a given location, calculated by the object recognition unit 150 along with the object's position, will also be simply referred to as "likelihood." The likelihood constitutes the accuracy of object recognition. The object recognition unit 150 may recognize objects for each type of object. Examples of object types include people, cars, buses, trucks, motorcycles, bicycles, buildings, and signs displayed on roads.

[0047] The weight adjustment unit 160 adjusts the weights for the first and second features, respectively, performed by the feature fusion unit 130, based on the likelihood calculated by the object recognition unit 150. The weight adjustment unit 160 can adjust the weights for the first and second features by determining the weights for the first and second features performed by the feature fusion unit 130, based on the likelihood calculated by the object recognition unit 150.

[0048] The weight adjustment unit 160 may include a learning model that outputs weights for the first feature and the second signal, respectively, in response to the likelihood input calculated by the object recognition unit 150. This learning model can be trained by reinforcement learning, which rewards improvement in the likelihood calculated by the object recognition unit 150. Hereinafter, the learning model included in the weight adjustment unit 160 will also be simply referred to as the "learning model".

[0049] When the learning model of the weight adjustment unit 160 is being trained, models that may be included in the feature fusion unit 130, the specific feature extraction unit 140, and the object recognition unit 150 may be excluded from the training.

[0050] The weight adjustment unit 160 may adjust the weights for the first and second features by the feature fusion unit 130 based on at least one of the likelihood and the environmental information described later. In this case, the weight adjustment unit 160 may include a learning model that outputs weights for the first feature and the second signal, respectively, for at least one of the inputs of the likelihood and the environmental information described later.

[0051] The operation of the inference device 1 will be explained.

[0052] Figure 3 is a flowchart showing the operation of the inference device 1. This flowchart can be executed by the control unit 100 according to a program.

[0053] The control unit 100 acquires image data and point cloud data from the camera 600 and LiDAR 700, respectively (S101).

[0054] The control unit 100 extracts a first feature from the image data and a second feature from the point cloud data (S102).

[0055] The control unit 100 weights the first feature and the second feature by the respective weights inferred by the learned learning model and fuses them (S103).

[0056] The control unit 100 extracts specific features from the features fused in step S103 (S104).

[0057] The control unit 100 infers the position and likelihood of an object using specific features (S105).

[0058] The control unit 100 learns a learning model through reinforcement learning, which rewards improvement in likelihood (S106).

[0059] The control unit 100 determines whether there is a next frame for the image data and point cloud data (S107). If it determines that there is no next frame (S107: NO), it terminates the process.

[0060] If the control unit 100 determines that there is a next frame (S107: YES), it executes step S101.

[0061] According to this embodiment, in response to a decrease in the object detection sensitivity based on image data or point cloud data due to changes in the detection environment of the camera 600 and LiDAR 700 over time, the weights of the features extracted from the image data and point cloud data are adjusted. Then, the object is recognized using a specific feature obtained by fusing the weighted features with the adjusted weights. This makes it possible to suppress the decrease in object detection sensitivity due to fluctuations in the detection environment in multimodal sensing.

[0062] (First embodiment) Figure 4 is an explanatory diagram illustrating a first embodiment of the inference device 1 according to the first embodiment. As shown in Figure 4, the inference device 1 is mounted, for example, on the upper part of the front windshield of a vehicle 900 on the vehicle side. In this case, the camera 600 and LiDAR 700 may be mounted so that the area in front of the vehicle 900 is within their respective detection ranges. The camera 600 and LiDAR 700 may be mounted on the vehicle 900 so that the optical axis of the camera 600 and the direction of the laser beam irradiation of the LiDAR 700 are in the same direction. By mounting the inference device 1 on the vehicle 900, recognizing a person as an object using the inference device 1, and displaying the recognized person on an image based on image data displayed on the vehicle 900's display, the user can easily recognize pedestrians, etc., regardless of changes in the environment. The inference device 1 displays the recognized person in a way that is recognizable to the driver, for example, by enclosing it with a rectangle on the image displayed on the vehicle 900's display. The inference device 1 may also display the recognized person in a way that is recognizable to the driver on an image created from point cloud data displayed on the vehicle 900's display.

[0063] (Second example) Figure 5 is an explanatory diagram illustrating a second embodiment of the inference device 1. As shown in Figure 5, the camera 600, LiDAR 700, and thermal camera 800 of the inference device 1 can be installed so that a common no-entry zone becomes the detection range for each. The inference device 1 extracts feature quantities from the signals output by the camera 600, LiDAR 700, and thermal camera 800, weights and fuses them, extracts specific feature quantities from the fused feature quantities, and uses these specific feature quantities to recognize an object. The object is, for example, a vehicle 900. This allows for easy detection of an object entering the no-entry zone regardless of changes in the environment.

[0064] (Second Embodiment) A second embodiment will now be described. The differences between this embodiment and the first embodiment are as follows. In the first embodiment, the weights for the first and second features are adjusted based on the likelihood calculated by the object recognition unit 150 along with the object recognition result. On the other hand, in this embodiment, the weights for the first and second features are adjusted based on environmental information corresponding to the detection environment in multimodal sensing. In all other respects, this embodiment is the same as the first embodiment, so redundant explanations will be omitted or simplified.

[0065] Figure 6 is a diagram showing the schematic configuration of the inference device 1. The functions of the control unit 100 will be explained with reference to Figure 6.

[0066] The camera encoder 110 can extract a first feature from the image data output from the camera 600.

[0067] The LiDAR encoder 120 can extract a second feature from the point cloud data output from the LiDAR 600.

[0068] The feature fusion unit 130 weights the first feature and the second feature by weights adjusted by the weight adjustment unit 160. The feature fusion unit 130 then fuses the weighted first feature and the second feature.

[0069] The specific feature extraction unit 140 extracts specific features from the features merged by the feature fusion unit 130 that the object recognition unit 150 uses to recognize objects.

[0070] The object recognition unit 150 recognizes an object using specific features.

[0071] The weight adjustment unit 160 adjusts the weights for the first and second features based on environmental information corresponding to the environment when the camera 600 and LiDAR 700 detect light and distance as image data and point cloud data, respectively. The weight adjustment unit 160 can adjust the weights for the first and second features by determining the weights for the first and second features based on the environmental information.

[0072] Environmental information includes, for example, time, weather, and date. When the environmental information is time, it corresponds to the brightness of the detection environment. The brightness of the detection environment affects the clarity of objects in image data and point cloud data. For example, objects in an image may be unclear in the dark of night. For example, in point cloud data, reflected light from direct sunlight hitting an object in bright daylight may be a disturbance, potentially causing missing points in the point cloud corresponding to that object.

[0073] Figure 7A shows an image of data captured by camera 600 at night. Figure 7B shows an image of point cloud data detected by LiDAR 700 at night. The image of data shown in Figure 7A and the image of point cloud data shown in Figure 7B have almost the same detection range. Comparing Figure 7A and Figure 7B, it can be seen that at night, the clarity of detected objects is higher with point cloud data than with image data.

[0074] The weight adjustment unit 160 can determine the weights for the first and second features based on the environmental information, using a table that defines the correspondence between environmental information and the weights for the first and second features, respectively. For example, if the environmental information is time, the table may define the weights for the first and second features as 0.3 and 0.7, respectively, corresponding to the environmental information for 18:00 to 5:00, which corresponds to nighttime. This ensures that during nighttime, object detection is performed with more emphasis on the second feature than the first feature. Alternatively, the table may define the weights for the first and second features as 0.7 and 0.3, respectively, corresponding to the environmental information for 6:00 to 17:00, which corresponds to daytime. This ensures that during daytime, object detection is performed with more emphasis on the first feature than the second feature.

[0075] According to this embodiment, the weights of the features extracted from the image data and point cloud data are adjusted in response to changes in the detection environment of the camera 600 and LiDAR 700 due to the passage of time, etc. Then, the object is recognized using a specific feature obtained by fusing the weighted features with the adjusted weights. This makes it possible to suppress the decrease in object detection sensitivity due to fluctuations in the detection environment in multimodal sensing.

[0076] (Third embodiment) A third embodiment will now be described. The differences between this embodiment and the first embodiment are as follows. In the first embodiment, the weights assigned to each of the image data and point cloud data are inferred by a learning model based on likelihood, and the learning model is trained by reinforcement learning, which rewards improvement in object recognition accuracy. On the other hand, in this embodiment, the weights assigned to each of the image data and point cloud data are inferred by a learning model based on environment-related information. The learning model is then trained by supervised learning using training data consisting of combinations of image data, point cloud data, and the correct object recognition results. In all other respects, this embodiment is the same as the first embodiment, so redundant explanations will be omitted or simplified.

[0077] Figure 8 shows a schematic configuration of the inference device 1. The functions of the control unit 100 will be explained with reference to Figure 8.

[0078] The camera encoder 110 can extract a first feature from the image data output from the camera 600.

[0079] The LiDAR encoder 120 can extract a second feature from the point cloud data output from the LiDAR 700.

[0080] The feature fusion unit 130 weights the first feature and the second feature by the respective weights adjusted by the weight adjustment unit 160. The feature fusion unit 130 then fuses the weighted first feature and the second feature.

[0081] The specific feature extraction unit 140 extracts specific features from the features merged by the feature fusion unit 130 that the object recognition unit 150 uses to recognize objects.

[0082] The object recognition unit 150 recognizes an object using specific features.

[0083] The weight adjustment unit 160 adjusts the weights for the first and second features based on environmental information corresponding to the environment when the camera 600 and LiDAR 700 detect light and distance as image data and point cloud data, respectively. As will be described later, the weight adjustment unit 160 can adjust the weights for the first and second features by inferring the weights for the first and second features using a learning model based on the environmental information.

[0084] Environmental information includes, for example, time, weather, and date. When the environmental information is time, it corresponds to the brightness of the detection environment. The brightness of the detection environment affects the clarity of objects in image data and point cloud data. For example, objects in an image may be unclear in the dark of night. For example, in point cloud data, reflected light from direct sunlight hitting an object in bright daylight may be a disturbance, potentially causing missing points in the point cloud corresponding to that object.

[0085] The weight adjustment unit 160 may include a learning model that outputs weights for the first and second features, respectively, in response to the input of environmental information. This learning model can be trained using training data consisting of image data, point cloud data, and the correct object recognition results. Specifically, the learning model can be trained as follows: Image data and point cloud data, which are the training data, are input to the camera encoder 110 and the LiDAR encoder 120, respectively, and the object recognition result recognized by the object recognition unit 150 is compared with the object recognition result of the training data. Then, the learning model is trained by backpropagation so that the difference between the object recognition result recognized by the object recognition unit 150 and the object recognition result of the training data becomes zero.

[0086] When the learning model of the weight adjustment unit 160 is being trained, models that may be included in the feature fusion unit 130, the specific feature extraction unit 140, and the object recognition unit 150 may be excluded from the training.

[0087] The weight adjustment unit 160 can use the learned learning model to infer weights for the first and second features based on environmental information.

[0088] The operation of the inference device 1 will be explained.

[0089] Figure 9 is a flowchart showing the operation of the inference device 1 during inference. This flowchart can be executed by the control unit 100 according to a program.

[0090] The control unit 100 acquires image data and point cloud data from the camera 600 and LiDAR 700, respectively (S201).

[0091] The control unit 100 extracts a first feature from the image data and a second feature from the point cloud data (S202).

[0092] The control unit 100 weights the first feature and the second feature by the respective weights inferred by the learned learning model and fuses them (S203).

[0093] The control unit 100 extracts specific features from the features fused in step S203 (S204).

[0094] The control unit 100 infers the position and likelihood of an object using specific features (S205).

[0095] The control unit 100 determines whether there is a next frame for the image data and point cloud data (S206). If it determines that there is no next frame (S206: NO), it terminates the process.

[0096] If the control unit 100 determines that there is another frame (S206:YES), it executes step S201.

[0097] Figure 10 is a flowchart showing the operation of the weight adjustment unit 160 of the inference device 1 during the training of the learning model. This flowchart can be executed by the control unit 100 according to a program.

[0098] The control unit 100 acquires training data consisting of combinations of image data and point cloud data with the correct object recognition results (S301).

[0099] The control unit 100 calculates the difference between the position of the object inferred based on the training data, which is image data and point cloud data, and the position of the object, which is the ground truth data (S302).

[0100] The learning model of the weight adjustment unit 160 is trained by backpropagation so that the difference calculated in step S302 becomes 0 (S302).

[0101] The control unit 100 determines whether there is other training data (S303). If the control unit 100 determines that there is no other training data (S303: NO), it terminates the process.

[0102] If the control unit 100 determines that there is other training data (S303: YES), it executes step S301.

[0103] (Fourth Embodiment) A fourth embodiment will now be described. The differences between this embodiment and the fourth embodiment are as follows. In the first embodiment, the weights to be assigned to the image data and point cloud data are inferred by a learning model based on likelihood, and the learning model is trained by reinforcement learning, which rewards improvement in object recognition accuracy. On the other hand, in this embodiment, the task is to move the robot to a destination. The task is performed by searching for a movement path for the robot based on the object recognition result and moving the robot along the obtained movement path. The weights to be assigned to the image data and point cloud data are inferred by a learning model based on the evaluation result of the task. The learning model is then trained by reinforcement learning, which rewards improvement in the evaluation result of the task. In all other respects, this embodiment is the same as the first embodiment, so redundant explanations will be omitted or simplified.

[0104] Figure 11 is a diagram showing the schematic configuration of the inference device 1. The functions of the control unit 100 will be explained with reference to Figure 11.

[0105] The camera 600 and LiDAR 700 are mounted on a moving robot. Specifically, the camera 600 and LiDAR 700 can be mounted on the robot so that their detection range is forward, which is the direction of the robot's movement.

[0106] The camera encoder 110 can extract a first feature from the image data output from the camera 600.

[0107] The LiDAR encoder 120 can extract a second feature from the point cloud data output from the LiDAR 600.

[0108] The feature fusion unit 130 weights the first feature and the second feature using weights adjusted by the weight adjustment unit 160. The feature fusion unit 130 then fuses the weighted first feature and the second feature.

[0109] The specific feature extraction unit 140 extracts specific features from the features merged by the feature fusion unit 130 that the object recognition unit 150 uses to recognize objects.

[0110] The object recognition unit 150 recognizes an object using specific features.

[0111] The path search unit 170 searches for a path for the robot to move based on the object recognition results from the object recognition unit 150. Specifically, the path search unit 170 searches for a partial path in which the robot does not come into contact with the objects recognized by the object recognition unit 150.

[0112] The robot drive unit 180 controls the robot to move along the path discovered by the path search unit 170. Specifically, the robot drive unit 180 transmits a robot drive signal to the robot to move along the discovered path. The robot drive signal may include information on the direction of movement and the distance traveled.

[0113] The task evaluation unit 190 evaluates the task of moving the robot to its destination. Specifically, the task evaluation unit 190 evaluates the task by detecting, for example, the number of times the robot makes contact with an object and the time it takes for the robot to reach its destination. If the time it takes for the robot to reach its destination exceeds a predetermined threshold, it may correspond to a state in which the robot is unable to reach its destination. Contact between the robot and an object can be detected, for example, using an acceleration sensor attached to the robot.

[0114] The weight adjustment unit 160 adjusts the weights for the first and second features based on the task evaluation results by the task evaluation unit 190. The weight adjustment unit 160 may include a learning model that outputs weights for the first feature and the second signal, respectively, in response to the task evaluation results from the task evaluation unit 190. This learning model can be learned by reinforcement learning, which rewards improvement in the task evaluation results. Specifically, for example, this learning model is learned by reinforcement learning designed so that the reward increases as the number of contacts between the robot and the object decreases until the robot reaches the destination, and the reward increases as the time to reach the destination decreases.

[0115] The weight adjustment unit 160 can use the learned learning model to infer weights for the first and second features based on environmental information.

[0116] The operation of the inference device 1 will be explained.

[0117] Figure 12 is a flowchart showing the operation of the inference device 1. This flowchart can be executed by the control unit 100 according to a program.

[0118] The control unit 100 acquires image data and point cloud data from the camera 600 and LiDAR 700, respectively (S401).

[0119] The control unit 100 extracts a first feature from the image data and a second feature from the point cloud data (S402).

[0120] The control unit 100 weights the first feature and the second feature by the respective weights inferred by the learned learning model and fuses them (S403).

[0121] The control unit 100 extracts specific features from the features fused in step S203 (S404).

[0122] The control unit 100 infers the position of the object using specific features (S405).

[0123] The control unit 100 searches for a path for moving the robot based on the object's position inferred in step S405 (S406).

[0124] The control unit 100 controls the robot to move along the searched path (S407).

[0125] The control unit 100 evaluates the task of moving the robot to its destination (S408). Step S408 may be performed after the robot has moved to its destination or after a predetermined time has elapsed since the robot control started.

[0126] The control unit 100 learns a learning model through reinforcement learning, which rewards improvement in the evaluation results of the task (S409).

[0127] The control unit 100 determines whether there is a next frame for the image data and point cloud data (S410). If it determines that there is no next frame (S410: NO), it terminates the process.

[0128] If the control unit 100 determines that there is another frame (S410: YES), it executes step S401.

[0129] Each of the embodiments described above has the following effects.

[0130] Features are extracted from two different signals, weighted with adjusted weights, and fused to form a fused set of features. A specific feature used for object recognition is then extracted from this fused feature, and the object is recognized using this specific feature. The weights are adjusted based on the object recognition accuracy, the task evaluation results based on the object recognition results, or environmental information corresponding to the environment. This suppresses the decrease in detection sensitivity due to fluctuations in the detection environment in multimodal sensing.

[0131] Furthermore, the first signal is used as two-dimensional image data, and the second signal is used as point cloud data obtained from LiDAR. This makes it possible to more effectively suppress the decrease in detection sensitivity due to fluctuations in the detection environment in multimodal sensing.

[0132] Furthermore, specific features are used to recognize the location of objects. This allows the specific location of an object to be displayed to the user.

[0133] Furthermore, the object recognition results are output along with the object recognition accuracy, and the weights for the first and second features are determined based on the object recognition accuracy. This makes it easier to suppress the decrease in detection sensitivity due to fluctuations in the detection environment in multimodal sensing.

[0134] Furthermore, the system includes a learning model that outputs weights for a first feature and a second feature in response to at least one input of object recognition accuracy and the aforementioned environmental information. The learning model is trained using reinforcement learning, which rewards improvement in object recognition accuracy. This improves the ability to suppress the decrease in detection sensitivity due to fluctuations in the detection environment in multimodal sensing.

[0135] Furthermore, a table is used that defines the correspondence between environmental response information and the weights for the first and second features, respectively. Based on the environmental response information, the weights for the first and second features are determined. This makes it easier and more flexible to suppress the decrease in detection sensitivity due to fluctuations in the detection environment in multimodal sensing.

[0136] Furthermore, the system includes a learning model that outputs weights for the first and second features in response to environmental information input. The learning model is trained using training data consisting of combinations of the first and second signals and the correct object recognition results. This improves the ability to suppress the decrease in detection sensitivity due to fluctuations in the detection environment in multimodal sensing.

[0137] The present invention is not limited to the embodiments described above.

[0138] For example, in this embodiment, some or all of the processing performed by the program may be replaced by hardware such as circuits.

[0139] Furthermore, the weights for the first and second features may be adjusted based on any two combinations of the following: object recognition accuracy, task evaluation results based on object recognition results, and environmental information corresponding to the environment.

[0140] While embodiments of the present invention have been described and illustrated in detail, the disclosed embodiments are for illustrative purposes only and are not limiting. The scope of the present invention should be interpreted in accordance with the language of the appended claims. [Explanation of Symbols]

[0141] 1 reasoning device, 100 Control unit, 110 Camera Encoder, 120 LiDAR encoders, 130 Feature Fusion Unit, 140 Specific feature extraction unit, 150 Object recognition section, 160 Weight adjustment section, 200 storage section, 300 display, 400 Input section, 500 Communications Department, 600 cameras, 700 LiDAR, 800 thermal cameras, 900 vehicles.

Claims

1. A first feature extraction unit that extracts a first feature from a first signal, A second feature extraction unit that extracts a second feature from the second signal, A feature fusion unit that weights and fuses the aforementioned first feature and the aforementioned second feature, A specific feature extraction unit extracts specific features used for object recognition from the fused features, A recognition unit that recognizes the object using the aforementioned specific feature quantities, A weight adjustment unit adjusts the weights used by the feature fusion unit to weight the first feature and the second feature, based on the recognition accuracy of the object by the recognition unit, the evaluation result of the task based on the recognition result of the object by the recognition unit, or environmental information corresponding to the environment when the first signal and the second signal are detected. An inference device having the following features.

2. The first signal is two-dimensional image data, The second signal is three-dimensional point cloud data obtained from LiDAR. The inference device according to claim 1.

3. The recognition unit recognizes the position of the object using the specific feature quantity. The inference device according to claim 1.

4. The recognition unit outputs the recognition result of the object along with the recognition accuracy of the object. The inference apparatus according to claim 1, wherein the weight adjustment unit determines the weights for the first feature and the second feature, respectively, based on the recognition accuracy of the object output by the recognition unit.

5. The weight adjustment unit has a learning model that outputs weights for the first feature and the second feature, respectively, in response to the input of at least one of the object recognition accuracy by the recognition unit and the environment-related information. The inference device according to claim 4, wherein the learning model is learned by reinforcement learning, which rewards the improvement in the accuracy of object recognition by the recognition unit.

6. The inference apparatus according to claim 1, wherein the weight adjustment unit determines the weights for the first feature and the second feature based on the environment information, using a table that defines the correspondence between the environment information and the weights for the first feature and the second feature, respectively.

7. The weight adjustment unit has a learning model that outputs weights for the first feature and the second feature, respectively, in response to the input of the environment-related information. The inference device according to claim 4, wherein the learning model is trained using training data of combinations of the first signal, the second signal, and the correct recognition result of the object.

8. A learning device having a control unit that trains the learning model described in claim 5 by reinforcement learning, with the improvement of the object recognition accuracy by the recognition unit as the reward.

9. A learning device having a control unit that trains the learning model described in claim 7 using training data of combinations of the first signal, the second signal, and the correct recognition result of the object.

10. Step (a1) of extracting a first feature from the first signal, Step (b1) of extracting a second feature from the second signal, Step (c1) involves weighting and merging the aforementioned first feature and the aforementioned second feature, Step (d1) involves extracting specific features to be used for object recognition from the fused features, The steps include: (e1) recognizing the object using the aforementioned specific features, Step (c1) is a step (f1) in which the weights used to weight the first feature and the second feature are adjusted based on the recognition result of the object recognized in step (e1), the evaluation result of the task based on the recognition result of the object in step (e1), or environmental information corresponding to the environment when the first signal and the second signal are detected. An inference method having

11. Step (a2) of extracting a first feature from the first signal, Step (b2) of extracting a second feature from the second signal, Step (c2) involves weighting and merging the aforementioned first feature and the aforementioned second feature, Step (d2) involves extracting specific features to be used for object recognition from the fused features, The steps include: (e2) recognizing the object using the aforementioned specific features, Step (c2) is a step (f2) in which the weights used to weight the first feature and the second feature are adjusted based on the recognition result of the object recognized in step (e2), the evaluation result of the task based on the recognition result of the object in step (e2), or environmental information corresponding to the environment when the first signal and the second signal are detected. An inference program that causes a computer to perform a process that has the following characteristics.

12. A learning method comprising the step (a3) ​​of training the learning model described in claim 5 by reinforcement learning, in which the improvement of the object recognition accuracy by the recognition unit is used as a reward.

13. A learning program for causing a computer to perform a process having step (a4) of training the learning model described in claim 5 by reinforcement learning, with the improvement of the recognition accuracy of the object by the recognition unit as the reward.

14. A learning method comprising the step (a5) of training the learning model described in claim 7 using training data of combinations of the first signal, the second signal, and the correct recognition result of the object.

15. A learning program for causing a computer to perform a process that includes the step (a6) of training the learning model described in claim 7 using training data of combinations of the first signal, the second signal, and the correct recognition result of the object.

Images