Object recognition device, object recognition method, and program

Abstract

To provide an object recognition device, an object recognition method, and a program capable of appropriately performing object detection.SOLUTION: An object recognition device includes: an acquisition section that acquires image data captured by a first imaging section that captures visible light and image data captured by a second imaging section that captures light other than visible light; a first object detection section that detects a position and a category of an object shown in the image data captured by the first imaging section; a second object detection section that detects a position and a category of the object shown in the image data captured by the second imaging section; and an integration section that obtains an integrated detection result that is an identification result of the position and the category of the object on the basis of the position and the category of the object detected by the first object detection section and the position and the category of the object detected by the second object detection section.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to an object recognition device, an object recognition method, and a program. [Background technology]

[0002] 2. Description of the Related Art Techniques for capturing images of a subject using a plurality of types of cameras with different systems are known.

[0003] For example, Patent Document 1 below discloses a depth acquisition device that can accurately acquire the depth to a subject, which is equipped with a visible camera, an infrared camera, and an infrared light source, and comprehensively calculates the depth from the results of depth estimation using the visible camera and the results of depth estimation using the infrared camera and the infrared light source. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2020-52001 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, the depth acquisition device described in Patent Document 1 above can only detect depth from the results of depth estimation using a visible camera and an infrared camera, and is unable to detect the position of an object in the image, nor can it determine the category of the object.

[0006] In view of the above-mentioned problems, the present disclosure aims to provide an object recognition device, an object recognition method, and a program that can appropriately perform object detection. [Means for solving the problem]

[0007] In order to solve the above-mentioned problems and achieve the object, the object recognition device of the present disclosure includes an acquisition unit that acquires image data captured by a first imaging unit that captures visible light and image data captured by a second imaging unit that captures light other than visible light, a first object detection unit that detects the position and category of an object that appears in the image data captured by the first imaging unit, a second object detection unit that detects the position and category of an object that appears in the image data captured by the second imaging unit, and an integration unit that obtains an integrated detection result that is a result of identifying the position and category of an object based on the position and category of the object detected by the first object detection unit and the position and category of the object detected by the second object detection unit.

[0008] In order to solve the above-mentioned problems and achieve the objectives, the object recognition method of the present disclosure includes the steps of acquiring image data captured by a first imaging unit that captures visible light and image data captured by a second imaging unit that captures light other than visible light, detecting the position and category of an object appearing in the image data captured by the first imaging unit, detecting the position and category of an object appearing in the image data captured by the second imaging unit, and obtaining an integrated detection result that is a result of identifying the position and category of the object based on the position and category of the object detected from the image data captured by the first imaging unit and the position and category of the object detected from the image data captured by the second imaging unit.

[0009] In order to solve the above-mentioned problems and achieve the objectives, the program of the present disclosure causes a computer to execute the following steps: acquiring image data captured by a first imaging unit that captures visible light and image data captured by a second imaging unit that captures light other than visible light; detecting the position and category of an object that appears in the image data captured by the first imaging unit; detecting the position and category of an object that appears in the image data captured by the second imaging unit; and obtaining an integrated detection result that is a result of identifying the position and category of the object based on the position and category of the object detected from the image data captured by the first imaging unit and the position and category of the object detected from the image data captured by the second imaging unit. [Effects of the Invention]

[0010] According to the present disclosure, it is possible to provide an object recognition device, an object recognition method, and a program that can appropriately perform object detection. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating an overview of an object recognition device according to the present disclosure. [Figure 2] FIG. 2 is a diagram illustrating an example of the configuration of an object recognition device according to the present disclosure. [Figure 3] FIG. 3 is a diagram illustrating an example of information stored in the image data storage unit of the object recognition device according to the present disclosure. [Figure 4] FIG. 4 is a diagram illustrating an example of information stored in the model storage unit of the object recognition device according to the present disclosure. [Figure 5] FIG. 5 is a diagram illustrating the processing of the integration unit of the object recognition device according to the present disclosure. [Figure 6] FIG. 6 is a flowchart showing the flow of the object recognition method according to the present disclosure. [Figure 7] FIG. 7 is a diagram illustrating an example configuration of an object recognition system according to the present disclosure. [Figure 8] FIG. 8 is a diagram illustrating an example of the configuration of an imaging device according to the present disclosure. [Figure 9] FIG. 9 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of the object recognition device according to the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. However, the present disclosure is not limited to the embodiments described below.

[0013] (Overview of the object recognition device) First, an overview of an object recognition device according to the present disclosure will be described using Fig. 1. Fig. 1 is a diagram illustrating an overview of an object recognition device according to the present disclosure. Fig. 1 shows an object recognition device 100 and objects O1 and O2 to be detected by the object recognition device 100.

[0014] As shown in FIG. 1, an object recognition device 100 according to the present disclosure is a device that detects objects. The object recognition device 100 may be mounted on various devices and used for object detection purposes. The category (type) of object detected by the object recognition device 100 may be any, and the object recognition device 100 may detect living objects such as people, or inanimate objects. The object detected by the object recognition device 100 may be, for example, a person, a ship, a forklift, an air vehicle, an artificial satellite, etc. It is preferable that the object recognition device 100 be capable of detecting objects of multiple categories, rather than just detecting objects of one category.

[0015] (Configuration of object recognition device) Next, the configuration of object recognition device 100 according to the present disclosure will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example configuration of object recognition device according to the present disclosure. As shown in Fig. 2, object recognition device 100 according to the present disclosure includes a communication unit 110, a storage unit 120, a control unit 130, a first imaging unit 140, a second imaging unit 150, and a display unit 160. These components will be described in order below.

[0016] The communication unit 110 is responsible for transmitting and receiving various types of information to and from external devices via wired or wireless connections. In the wired case, it may be realized by, for example, a wired LAN (Local Area Network) terminal or a NIC (Network Interface Card) equipped with various interfaces such as a USB (Universal Serial Bus) terminal. In the wireless case, it may be realized by a wireless LAN defined by IEEE802.11, Bluetooth (registered trademark), Wi-Fi (registered trademark), or the like.

[0017] The storage unit 120 is a storage device that stores various types of information. The storage unit 120 includes a main storage device and an auxiliary storage device. The main storage device may be realized by a semiconductor memory element such as a random access memory (RAM), a read only memory (ROM), or a flash memory. The auxiliary storage device may be realized by a hard disk or a solid state drive (SSD), for example.

[0018] The image data storage unit 121 stores information related to image data. An example of information stored in the image data storage unit 121 will now be described with reference to Fig. 3. Fig. 3 is a diagram illustrating an example of information stored in the image data storage unit of the object recognition device according to the present disclosure.

[0019] As shown in FIG. 3, the image data storage unit 121 stores information relating to the items "image data ID," "image data," "type," and "image capture date and time."

[0020] The "image data ID" is an identifier that identifies the image data and is represented by a character string, a number, or the like. The "image data" is image data identified by the "image data ID" and may be image data in a file format such as JPEG (Joint Photographic Experts Group) or TIFF (Tag Image File Format). The "type" is information that indicates the type of image data identified by the "image data ID" and is represented, for example, by a visible light image or an infrared image. The "image capture date and time" is information that indicates the date and time when the image data identified by the "image data ID" was captured.

[0021] That is, Figure 4 shows an example in which the type of image data "IMGDT#1" identified by the image data ID "IMGID#1" is "TYP#1" and the image capture date and time of the image data is stored as "TM#1".

[0022] The information stored in the image data storage unit 121 is not limited to information relating to the items "image data ID," "image data," "type," and "date and time of image capture," but may also store information relating to any other image data.

[0023] The model storage unit 122 stores information about a machine learning model used for object detection. An example of the information stored in the model storage unit 122 will now be described with reference to Fig. 4. Fig. 4 is a diagram illustrating an example of the information stored in the model storage unit of the object recognition device according to the present disclosure.

[0024] As shown in FIG. 4, the model storage unit 122 stores information relating to the items "model ID" and "model data."

[0025] A "model ID" is an identifier that identifies a machine learning model and is represented by a string or number. "Model data" is data for the machine learning model identified by the "model ID." The machine learning model may be configured using a deep neural network (DNN) or a convolutional neural network (CNN). "Model data" includes various information, such as connection information about how nodes included in each of the multiple layers that make up the neural network are connected to each other, and connection coefficients that are multiplied by numerical values ​​input and output between connected nodes.

[0026] That is, FIG. 4 shows an example in which model data "MDDT#1" of a model identified by a model ID "MDID#1" is stored.

[0027] Note that the information stored in the model storage unit 122 is not limited to information relating to the items "model ID" and "model data", and may store information relating to any other machine learning model.

[0028] The first imaging unit 140 captures images using a visible light camera that captures visible light. The first imaging unit 140 may be a visible light camera that includes optical elements and an imaging element. The optical elements are elements that constitute an optical system, such as lenses, mirrors, prisms, and filters. The imaging element is an element that converts light that has entered through the optical elements into an image signal, which is an electrical signal. The imaging element may be, for example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor.

[0029] The second imaging unit 150 captures images using a camera that captures light other than visible light. The second imaging unit 150 may be a camera that captures light of any wavelength band other than visible light. However, in this embodiment, image data is captured using an infrared camera that captures infrared light. The second imaging unit 150 may be realized, for example, by an MCT (Mercury Cadmium Telluride) optical sensor. The MCT optical sensor is an infrared detector using a semiconductor made of mercury (Hg), cadmium (Cd), and tellurium (Te), and has the characteristics of ultra-high sensitivity and ultra-fast response. Alternatively, the second imaging unit 150 may be realized by a dual-wavelength quantum dot infrared sensor using a quantum dot infrared photodetector (QDIP). The dual-wavelength quantum dot infrared sensor can detect both mid-infrared light emitted from high-temperature objects and far-infrared light emitted from low-temperature objects.

[0030] In this embodiment, the first imaging unit 140 and the second imaging unit 150 are arranged so that the imaging range of the first imaging unit 140 and the imaging range of the second imaging unit 150 overlap in at least a partial area. The first imaging unit 140 and the second imaging unit 150 may be installed in any position, but are preferably installed outdoors, for example.

[0031] The display unit 160 displays various types of information. For example, the display unit 160 may display a GUI (Graphical User Interface) for accepting operations related to various processes, the results of various processes, etc. The display unit 160 may be realized by a liquid crystal display, an organic EL (Electro Luminescence) display, a micro LED (Light Emitting Diode) display, etc.

[0032] The control unit 130 is a controller that executes various types of arithmetic processing and processing that realizes functions. The control unit 130 is realized by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like executing various programs stored in the storage unit 120 using RAM as a work area. The control unit 130 may also be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0033] As shown in FIG. 5, the control unit 130 includes an acquisition unit 131, a first object detection unit 132, a second object detection unit 133, a position determination unit 134, an integration unit 135, and a display control unit 136. The control unit 130 realizes these functions and performs these processes by reading and executing a program (software) from the storage unit 120. The control unit 130 may perform these processes using a single arithmetic unit such as a CPU, or may include multiple arithmetic units and perform the processes using the multiple arithmetic units. At least some of the acquisition unit 131, the first object detection unit 132, the second object detection unit 133, the position determination unit 134, the integration unit 135, and the display control unit 136 may be realized by hardware circuits. The program for the control unit 130 stored in the storage unit 120 may be stored in a recording medium readable by the object recognition device 100.

[0034] (Object recognition device processing) Next, the processing contents of the object recognition device 100 will be described.

[0035] (Image data acquisition) The acquisition unit 131 acquires various types of image data. Specifically, the acquisition unit 131 acquires image data captured by the first imaging unit 140. Hereinafter, the image (image data) captured by the first imaging unit 140 will be referred to as a visible light image (visible light image data) as appropriate. Each pixel of the visible light image represents the visible light received at the position of that pixel by the magnitude of each RGB brightness. Furthermore, the acquisition unit 114 acquires image data captured by the second imaging unit 150. Hereinafter, the image (image data) captured by the first imaging unit 140 will be referred to as an infrared image (infrared image data) as appropriate. Hereinafter, each pixel of the infrared image represents the intensity of infrared light received at the position of that pixel as a pixel value (specifically, brightness).

[0036] (Object detection) The first object detection unit 132 detects an object that appears in image data (visible light image data) captured by the first imaging unit 140. Specifically, the first object detection unit 132 executes an object detection process on the image data captured by the first imaging unit 140, and calculates the category (type) of the object that appears in the image data, the position of the object, and a reliability score of the object (an index indicating the reliability of the detection result). That is, from the image data captured by the first imaging unit 140, the first object detection unit 132 generates a bounding box that surrounds the detected object, coordinate values ​​that indicate the position of the object in the image, the name of the object category, and a numerical value of the object reliability score.

[0037] The object detection process by the first object detection unit 132 may use, for example, a trained model that has been machine-learned to determine the correspondence between the feature amounts of image data of an object and the category (ground truth label) of that object. For example, this may be realized by an object detection process model such as a convolutional neural network (CNN), a regular convolutional neural network (R-CNN), or Fast R-CNN. The trained model used by the first object detection unit 132 may be one that has learned the features of various objects captured in visible light images using visible light images capturing various objects as training data, in other words, training data. However, the present invention is not limited to using such a machine learning technique. The first object detection unit 132 may, for example, calculate the object category, object position, and object reliability score by comparing the visible light image data with reference image data.

[0038] The second object detection unit 133 detects an object that appears in image data (infrared image data) captured by the second imaging unit 150. Specifically, the second object detection unit 133 executes an object detection process on the image data captured by the second imaging unit 150, and calculates the category of the object that appears in the image data, the position of the object, and a reliability score of the object. That is, like the first object detection unit 132, the second object detection unit 133 generates, from the image data captured by the second imaging unit 150, a bounding box that surrounds the detected object, coordinate values ​​that indicate the position of the object in the image, the name of the object category, and a numerical value of the object's reliability score.

[0039] The object detection process by the second object detection unit 133 may be realized by the same method as the object detection process by the first object detection unit 132. Note that the trained model used in the second object detection unit 133 may be one in which infrared images in which various objects appear are used as training data, in other words, as training data, and the characteristics of various objects appearing in the infrared images are learned.

[0040] (Object identity determination) The position determination unit 134 determines whether the object detected by the first object detection unit 132 and the object detected by the second object detection unit 133 are the same, based on the position of the object detected by the first object detection unit 132 and the position of the object detected by the second object detection unit 133. Here, "the objects are the same" means that the detection result of the first object detection unit 132 and the detection result of the second object detection unit 133 are detection results for the same object. The position determination unit 134 determines that the objects are the same when the distance between the position of the object detected by the first object detection unit 132 and the position of the object detected by the second object detection unit 133 is within a predetermined distance range. On the other hand, the position determination unit 134 determines that the objects are not the same when the distance between the position of the object detected by the first object detection unit 132 and the position of the object detected by the second object detection unit 133 is outside the predetermined distance range. For example, the position determination unit 134 may calculate the distance between the center position of the object detected by the first object detection unit 132 and the center position of the object detected by the second object detection unit 133, and determine whether the calculated distance is within a predetermined distance range. The positions of the objects detected by the first object detection unit 132 and the second object detection unit 133 may be the positions of the objects in the coordinate system of the image, or may be the positions of the objects in the coordinate system of the space in which the object recognition device 100 is installed. The predetermined distance range may be set arbitrarily, but may be within 10 to 20 pixels, for example, when the position of the object in the coordinate system of the image is used.

[0041] (Generating Integrated Detection Results) The integrating unit 135 generates an integrated detection result, which is a result of identifying the position and category of an object, based on the object detection result of the first object detection unit 132 and the object detection result of the second object detection unit 133.

[0042] For example, the integrating unit 135 compares the detection result by the first object detection unit 132 and the detection result by the second object detection unit 133 for objects determined to be the same object (within a predetermined distance range), and adopts one of the detection results as the integrated detection result for the object based on a predetermined criterion. Any predetermined criterion may be used, but in this embodiment, a reliability score is used. That is, the integrating unit 135 adopts, for objects determined to be the same object by the position determination unit 134, the detection result with the larger reliability score between the detection result by the first object detection unit 132 and the detection result by the second object detection unit 133, as the integrated detection result for the object.

[0043] In addition, for an object detected by either the first object detection unit 132 or the second object detection unit 133 (an object detected by the first object detection unit 132 or the second object detection unit 133 but not determined to be the same object by the position determination unit 134), the integration unit 135 adopts the detection result of the object by either the first object detection unit 132 or the second object detection unit 133 as the integrated detection result for the object.

[0044] The processing of the integrating unit 135 will be described with reference to FIG. 5. FIG. 5 is a diagram illustrating the processing of the integrating unit of the object recognition device according to the present disclosure. In the example of FIG. 5, it is assumed that object detection processing is performed on visible light image data A and infrared image data B, and that an object Oa1 in the category of "ship" and with a reliability score of "90%" and an object Oa2 in the category of "ship" and with a reliability score of "60%" are detected from the visible light image data A. It is also assumed that an object Ob1 in the category of "car" and with a reliability score of "30%" and an object Ob2 in the category of "ship" and with a reliability score of "80%" are detected from the infrared image data B. It is assumed that objects Oa2 and Ob1 are determined to be the same object because they are detected at positions within a predetermined distance range.

[0045] In this case, the integrating unit 135 compares the reliability scores of the objects Oa2 and Ob1 determined to be the same object, and adopts the detection result of the object Oa2, which has the higher reliability score, as the integrated detection result Oc1 for that object. Furthermore, for object Oa1 detected from the visible light image data A at a position where no object was detected in the infrared image data B (i.e., an object not determined to be the same object), the integrating unit 135 adopts the detection result using the visible light image data A as the integrated detection result Oc2 for that object. Furthermore, for object Ob2 detected from the infrared image data B at a position where no object was detected in the visible light image data A, the integrating unit 135 adopts the detection result using the infrared image data B as the integrated detection result Oc3 for that object. The integrating unit 135 combines the integrated detection results of the objects, i.e., the integrated detection results Oc1, Oc2, and Oc3 in the example of FIG. 5, as the integrated detection result.

[0046] Note that the integrating unit 135 preferably adopts as the integrated detection result a detection result whose reliability score is equal to or greater than a predetermined threshold. For example, for an object detected by either the first object detection unit 132 or the second object detection unit 133, the integrating unit 135 adopts the detection result of that object as the integrated detection result if the reliability score of the detection result of that object by either the first object detection unit 132 or the second object detection unit 133 is equal to or greater than a predetermined threshold. For example, in FIG. 5, if the threshold is 85%, for object Oa1 detected only from visible light image data A, the reliability score of the detection result based on visible light image data A exceeds the threshold, so the detection result based on visible light image data A is adopted as the integrated detection result Oc2 for that object. On the other hand, for object Ob2 detected only from infrared image data B, the reliability score of the detection result based on infrared image data B is below the threshold, so the detection result based on infrared image data B is not adopted as the integrated detection result for that object. In other words, in this case, the integrated detection result does not include a detection result indicating that an object was detected at the position of object Ob2.

[0047] For example, the integration unit 135 may use a looser criterion for determining that the reliability score is equal to or greater than a threshold for an object detected by both the first object detection unit 132 and the second object detection unit 133 (an object determined to be the same) than the criterion for an object detected by either the first object detection unit 132 or the second object detection unit 133 (an object not determined to be the same).

[0048] Here, for example, the reliability score of the detection result with the larger reliability score between the detection result by the first object detection unit 132 and the detection result by the second object detection unit 133 for objects determined to be the same is set as the target reliability score. In this case, for example, the integrating unit 135 calculates a corrected reliability score by adding a predetermined value to the target reliability score (or multiplying it by a predetermined value greater than 1). Then, if the corrected reliability score is equal to or greater than a threshold, the integrating unit 135 adopts the detection result with the larger reliability score as the integrated detection result for that object. On the other hand, if the corrected reliability score is less than the threshold, the integrated detection result does not include a detection result indicating that an object was detected at that position. For example, in Figure 5, if the threshold is 85% and the predetermined value to be added to the target reliability score is 30%, the reliability score (target reliability score) of the detection result of object Oa2, which has the higher reliability score between object Oa2 and object Ob1 determined to be the same object, is 60%, so the corrected reliability score becomes 90%, which exceeds the threshold, and the detection result of object Oa2 is adopted as the integrated detection result Ob1.

[0049] Note that the method of loosening the determination criteria for objects determined to be identical is not limited to using a corrected reliability score with a higher reliability score. For example, the integrating unit 135 may set a lower threshold for the reliability score of the detection results of objects determined to be identical than the threshold for the reliability score of the detection results of objects not determined to be identical. Then, if the target reliability score is equal to or greater than the threshold, the integrating unit 135 may adopt the detection result as the integrated detection result for that object. For example, in FIG. 5, if the threshold for objects determined to be identical is 55%, the reliability score (target reliability score) of the detection result of object Oa2 is 60%, which exceeds the threshold, and therefore the detection result of object Oa2 is adopted as the integrated detection result Oc1.

[0050] (Output of integrated detection results) The display control unit 136 outputs the integrated detection result generated by the integration unit 135. The display control unit 136 may output the integrated detection result by any method, for example, by transmitting it to an external device via communication, or by displaying the integrated detection result on the display unit 160. In the example of FIG. 5, the display control unit 136 displays the integrated detection results for each object on the display unit 160 as an integrated detection result image C. For example, the display control unit 136 may display a bounding box surrounding the detected object, the position coordinate values ​​of the detected object, and the name of the category of the detected object superimposed on the image data. Furthermore, the display control unit 136 may display an object ID identifying the detected object, the position coordinate values ​​of the detected object, and the name of the category of the detected object side by side in a tabular format on the display unit 160.

[0051] According to the configuration of object recognition device 100 described above, it is possible to display an integrated detection result that integrates the object detection results for multiple images. Therefore, it is possible to provide object recognition device 100 that can appropriately detect an object that is difficult to detect using only one captured image.

[0052] (About object recognition methods) Next, an object recognition method according to the present disclosure will be described with reference to Fig. 6. Fig. 6 is a flowchart showing the flow of the object recognition method according to the present disclosure. Below, the object recognition method according to the present disclosure will be described along the flow shown in Fig. 6.

[0053] First, the object recognition device 100 acquires a visible light image using the first imaging unit 140 (step S101). Next, the object recognition device 100 detects an object appearing in the visible light image (step S102). Meanwhile, in parallel, the object recognition device 100 acquires an infrared light image using the second imaging unit 150 (step S103). Next, the object recognition device 100 detects an object appearing in the infrared light image (step S104).

[0054] The object recognition device 100 determines whether or not an object has been detected at the same position (whether or not the object detected from the visible light image and the object detected from the infrared light image are the same) (step S105). If objects are detected at the same position in step S105 (step S105: Yes), the object recognition device 100 adopts the detection result with the higher reliability score for the same position (step S106).

[0055] If no object is detected at the same position in step S105 (step S105: No), object recognition device 100 determines whether the reliability score of the object detected on one side is equal to or greater than a predetermined threshold (step S107). If the reliability score of the object detected on one side is equal to or greater than a predetermined threshold in step S107 (step S107: Yes), object recognition device 100 adopts the detection result (step S108).

[0056] In step S107, if the reliability score of one of the detected objects is not equal to or greater than the predetermined threshold (step S107: No), object recognition device 100 does not adopt the detection result (step S109).

[0057] When the processing of steps S105 to S109 is completed for all objects detected in the first captured image and the second captured image, the object recognition device 100 outputs a detection result image that integrates the detection results of all objects (step S110).

[0058] This makes it possible to perform object detection on each of the multiple images and display an integrated detection result that integrates the object detection results from both images. Therefore, it is possible to provide an object recognition method and program that can appropriately detect objects that are difficult to detect using only a single captured image.

[0059] (About object recognition systems) The object recognition device 100 described above is configured to include an imaging unit, but it may also be configured to acquire image data from an external device and use that image data to perform object detection. Specifically, the object recognition system 1 may be configured as shown in FIG. 7. FIG. 7 is a diagram showing an example configuration of an object recognition system according to the present disclosure. The object recognition system 1 includes the object recognition device 100, an imaging device 200, and a network N.

[0060] The object recognition device 100 may have the same configuration as the embodiment described above. Note that the object recognition device 100 may have a configuration in which the first imaging unit 140 and the second imaging unit 150 are removed from the embodiment described above.

[0061] The imaging device 200 is a device that captures various types of image data. The imaging device 200 may be any device equipped with a camera or an infrared camera, and may be, for example, a smartphone, a PDA (Personal Digital Assistant), a tablet terminal, or the like.

[0062] The network N is responsible for exchanging information between the object recognition device 100 and the image capture device 200. The network N may be realized wirelessly or wired. If it is wired, it may be realized by a wired LAN, a USB cable, or the like. If it is wireless, it may be realized by a wireless LAN, or the like.

[0063] The object recognition system 1 functions as a single system by the object recognition device 100 and the imaging device 200 exchanging information via a network N and each performing information processing.

[0064] (Configuration of imaging device) Next, an imaging device 200 according to the present disclosure will be described with reference to FIG. 8. FIG. 8 is a diagram illustrating an example configuration of an imaging device according to the present disclosure. As shown in FIG. 8, the imaging device 200 according to the present disclosure includes a communication unit 210, a storage unit 220, a control unit 230, a first imaging unit 240, and a second imaging unit 250. The communication unit 210, the storage unit 220, the first imaging unit 240, and the second imaging unit 250 may be the same as the communication unit 110, the storage unit 120, the first imaging unit 140, and the second imaging unit 150 of the object recognition device 100, respectively, and therefore description thereof will be omitted. Below, the control unit 230, which has a configuration different from that of the object recognition device 100, will be described.

[0065] The control unit 230 is a controller that executes various types of arithmetic processing and processes that realize functions. The control unit 230 is realized by a CPU, an MPU, or the like executing various programs stored in the storage unit 220 using RAM as a work area. The control unit 230 may also be realized by an integrated circuit such as an ASIC or FPGA.

[0066] 5, the control unit 230 includes a first acquisition unit 231, a second acquisition unit 232, and an output unit 233. The control unit 230 realizes these functions and executes these processes by reading and executing a program (software) from the storage unit 220.

[0067] The first acquisition unit 231 acquires image data captured by the first imaging unit 240. That is, the first acquisition unit 231 acquires visible image data captured by the visible camera of the first imaging unit 240. After acquiring the image data captured by the first imaging unit 240, the first acquisition unit 231 stores the acquired image data in the image data storage unit 221.

[0068] The second acquisition unit 232 acquires image data captured by the second imaging unit 250. That is, the second acquisition unit 232 acquires infrared image data captured by the infrared camera of the second imaging unit 250. After acquiring the image data captured by the second imaging unit 250, the second acquisition unit 232 stores the acquired image data in the image data storage unit 221.

[0069] The output unit 233 receives a request from an external device and outputs, in other words, transmits, image data to the external device that received the request. That is, the output unit 233 transmits the image data to the external device, for example, the object recognition device 100, via the communication unit 210 and over the network N.

[0070] As described above, the imaging device 200 can acquire image data captured using different types of imaging devices and provide the acquired image data to an external device. Therefore, even if the external device does not have an imaging device, by combining it with the imaging device 200, the external device can acquire image data.

[0071] (Hardware configuration) The object recognition device 100 according to the above-described embodiment is realized, for example, by a computer 1000 configured as shown in Fig. 9. Fig. 9 is a hardware configuration diagram showing an example of a computer that realizes the functions of the object recognition device according to the present disclosure. The computer 1000 is connected to an output device 1010 and an input device 1020, and has a configuration in which a calculation device 1030, a primary storage device 1040, a secondary storage device 1050, an output IF (Interface) 1060, an input IF 1070, and a network IF 1080 are connected via a bus 1090.

[0072] The arithmetic device 1030 operates based on programs stored in the primary storage device 1040 and secondary storage device 1050, programs read from the input device 1020, and the like, and executes various processes. The primary storage device 1040 is a memory device, such as a RAM, that temporarily stores data used by the arithmetic device 1030 for various calculations. The secondary storage device 1050 is a storage device that stores data used by the arithmetic device 1030 for various calculations and various databases, and is realized by a ROM, HDD, flash memory, or the like.

[0073] The output IF 1060 is an interface for transmitting information to be output to an output device 1010 that outputs various types of information, such as a monitor or a printer, and is realized by a connector conforming to a standard such as USB (Universal Serial Bus), DVI (Digital Visual Interface), or HDMI (High Definition Multimedia Interface), etc. The input IF 1070 is an interface for receiving information from various input devices 1020, such as a mouse, keyboard, scanner, etc., and is realized by a USB, etc.

[0074] The input device 1020 may be a device that reads information from, for example, an optical recording medium such as a CD (Compact Disc), a DVD (Digital Versatile Disc), or a PD (Phase Change Rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory. The input device 1020 may also be an external storage medium such as a USB memory.

[0075] The network IF 1080 receives data from other devices via the network N and sends it to the arithmetic device 1030, and also transmits data generated by the arithmetic device 1030 to other devices via the network N.

[0076] The arithmetic unit 1030 controls the output device 1010 and the input device 1020 via the output IF 1060 and the input IF 1070. For example, the arithmetic unit 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040 and executes the loaded program.

[0077] For example, when the computer 1000 functions as the object recognition device 100, the arithmetic unit 1030 of the computer 1000 realizes the functions of the control unit 130 of the object recognition device 100 by executing a program loaded onto the primary storage device 1040.

[0078] (Composition and Effects) The object recognition device 100 according to the first aspect includes an acquisition unit 131 that acquires image data captured by a first imaging unit 140 that captures visible light and image data captured by a second imaging unit 150 that captures light other than visible light; a first object detection unit 132 that detects the position and category of an object appearing in the image data captured by the first imaging unit 140; a second object detection unit 133 that detects the position and category of an object appearing in the image data captured by the second imaging unit 150; and an integration unit 135 that obtains an integrated detection result that identifies the position and category of an object based on the position and category of the object detected by the first object detection unit 132 and the position and category of the object detected by the second object detection unit 133.

[0079] This configuration makes it possible to display an integrated detection result that integrates the object detection results for multiple images, thereby providing an object recognition device 100 that can appropriately detect objects that are difficult to detect using only one captured image.

[0080] The object recognition device 100 according to the second aspect is the object recognition device 100 according to the first aspect, and further includes a position determination unit 134 that determines that the objects shown in the image data captured by the first imaging unit 140 and the object detected by the second object detection unit 133 are the same object when the positions of the objects are located within a predetermined distance range, and the integration unit 135 compares the detection results of the object by the first object detection unit 132 and the detection results of the object by the second object detection unit 133 that have been determined to be the same object, and adopts one of the two as the integrated detection result based on a predetermined criterion.

[0081] With this configuration, the detection results of the object detection results for the images captured by the visible light camera and the images captured by other cameras that match a predetermined scale can be used as the integrated detection result, making it possible to perform highly accurate object detection that is difficult to detect using only one captured image.

[0082] The object recognition device 100 according to the third aspect is the object recognition device 100 according to the second aspect, in which the first object detection unit 132 and the second object detection unit 133 calculate a reliability score indicating the reliability of the detection result, and the integration unit 135 adopts the detection result of the first object detection unit 132 or the detection result of the second object detection unit 133, whichever has the larger reliability score, as the integrated detection result for objects determined to be the same object by the position determination unit 134.

[0083] With this configuration, the most reliable detection result from the object detection results for images captured by the visible light camera and images captured by other cameras can be used as the integrated detection result, making it possible to perform highly accurate object detection that is difficult to detect using only one captured image.

[0084] The object recognition device 100 according to the fourth aspect is the object recognition device 100 according to any one of the first to third aspects, in which the integration unit 135 adopts, as the integrated detection result, the object detected by either the first object detection unit 132 or the second object detection unit 133, if the reliability score of the object detected by either unit exceeds a predetermined threshold.

[0085] This configuration makes it possible to detect an object that was not detected in one image from the other image, thereby enabling appropriate object detection that would be difficult to detect from only one captured image.

[0086] The object recognition device 100 according to the fifth aspect is the object recognition device 100 according to any one of the first to fourth aspects, and further includes a display control unit 136 that displays the integrated detection result on a display unit.

[0087] According to this configuration, it is possible to display the detection result of an object that is difficult to detect from only one captured image.

[0088] The object recognition device 100 according to a sixth aspect is the object recognition device 100 according to any one of the first to fifth aspects, in which the second imaging unit 150 captures an image of infrared light.

[0089] According to this configuration, the detection results from both the visible light camera and the infrared camera can be integrated, making it possible to appropriately detect objects that are difficult to detect using only one captured image.

[0090] The object recognition method according to the first aspect includes the steps of acquiring image data captured by a first imaging unit 140 that captures visible light and image data captured by a second imaging unit 150 that captures light other than visible light, detecting the position and category of an object appearing in the image data captured by the first imaging unit 140, detecting the position and category of an object appearing in the image data captured by the second imaging unit 150, and obtaining an integrated detection result that identifies the position and category of the object based on the position and category of the object detected from the image data captured by the first imaging unit 140 and the position and category of the object detected from the image data captured by the second imaging unit 150.

[0091] This configuration makes it possible to appropriately detect an object that is difficult to detect using only one captured image.

[0092] The program according to the first aspect causes a computer to execute the following steps: acquiring image data captured by a first imaging unit 140 that captures visible light and image data captured by a second imaging unit 150 that captures light other than visible light; detecting the position and category of an object appearing in the image data captured by the first imaging unit 140; detecting the position and category of an object appearing in the image data captured by the second imaging unit 150; and obtaining an integrated detection result that identifies the position and category of an object based on the position and category of the object detected from the image data captured by the first imaging unit 140 and the position and category of the object detected from the image data captured by the second imaging unit 150.

[0093] This configuration makes it possible to appropriately detect an object that is difficult to detect using only one captured image.

[0094] Although the embodiments of the present invention have been described above, the embodiments are not limited to the contents of these embodiments. Furthermore, the above-described components include those that can be easily imagined by a person skilled in the art, those that are substantially the same, and those that are within the scope of what is called equivalents. Furthermore, the above-described components can be combined as appropriate. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the spirit of the above-described embodiments. [Explanation of symbols]

[0095] 1. Object Recognition System 100 Object recognition device 110 Communications Department 120 Storage section 121 Image data storage unit 122 Model Memory Unit 130 control section 131 Acquisition Department 132 First object detection unit 133 Second object detection unit 134 Position determination section 135 Integration Department 136 Display control unit 140 First Imaging Unit 150 Second imaging unit 160 Display section 200 Imaging device 210 Communications Department 220 Storage section 221 Image data storage unit 230 Control Unit 231 First Acquisition Department 232 Second Acquisition Department 233 Output section 240 First Imaging Unit 250 Second Imaging Unit N Network

Claims

1. an acquisition unit that acquires image data captured by a first imaging unit that captures visible light and image data captured by a second imaging unit that captures light other than visible light; a first object detection unit that detects the position and category of an object captured in image data captured by the first imaging unit; a second object detection unit that detects the position and category of an object appearing in image data captured by the second imaging unit; an integration unit that obtains an integrated detection result that is a result of identifying the position and category of an object based on the position and category of the object detected by the first object detection unit and the position and category of the object detected by the second object detection unit; Including, Object recognition device.

2. a position determination unit that determines that an object captured in image data captured by the first imaging unit and an object detected by the second object detection unit are the same object when the positions of the objects are located within a predetermined distance range, the integrating unit compares the object detection result by the first object detection unit and the object detection result by the second object detection unit, which are determined to be the same object, and adopts one of the two as the integrated detection result based on a predetermined criterion. The object recognition device according to claim 1 .

3. the first object detection unit and the second object detection unit calculate a reliability score indicating a reliability of the detection result; the integrating unit adopts, as the integrated detection result, a detection result of the object having a larger reliability score out of the detection result of the first object detection unit and the detection result of the second object detection unit, for objects determined to be the same object by the position determining unit. The object recognition device according to claim 2 .

4. When a reliability score of an object detected by one of the first object detection unit and the second object detection unit exceeds a predetermined threshold, the integration unit adopts the object detected by the other unit as the integrated detection result. The object recognition device according to any one of claims 1 to 3.

5. a display control unit that displays the integrated detection result on a display unit; The object recognition device according to any one of claims 1 to 3.

6. The second imaging unit captures infrared light. The object recognition device according to any one of claims 1 to 3.

7. acquiring image data captured by a first imaging unit that captures visible light and image data captured by a second imaging unit that captures light other than visible light; Detecting a position and a category of an object captured in image data captured by the first imaging unit; Detecting a position and a category of an object captured in image data captured by the second imaging unit; and obtaining an integrated detection result, which is a result of identifying the position and category of the object, based on the position and category of the object detected from the image data captured by the first imaging unit and the position and category of the object detected from the image data captured by the second imaging unit. Object recognition method.

8. acquiring image data captured by a first imaging unit that captures visible light and image data captured by a second imaging unit that captures light other than visible light; Detecting a position and a category of an object captured in image data captured by the first imaging unit; Detecting a position and a category of an object captured in image data captured by the second imaging unit; obtaining an integrated detection result that is a result of identifying the position and category of the object based on the position and category of the object detected from the image data captured by the first imaging unit and the position and category of the object detected from the image data captured by the second imaging unit; A program that causes a computer to execute the following.

Images