Object detection method, program, and object detection system

Abstract

In the object detection method, a first image and a second image are acquired, each pixel of the second image and each pixel of the first image are in one-to-one correspondence (step S1), a first recognition process for recognizing a type of a first object reflected in the first image is executed (step S2), and a second recognition process for recognizing a type of a second object reflected in the second image is executed (step S2). And a second recognition process for recognizing the position of a second object reflected in the second image (step S3), and executing a detection process for detecting the first object and the second object as the same object when a first region in the first image, which is based on the first object, and a second region in the second image, which is based on the second object, overlap (step S4).

Description

Technical Field

[0001] The present disclosure relates to an object detection method, a program, and an object detection system. Background Art

[0002] For example, Patent Document 1 discloses an object recognition device that recognizes two objects detected using a radar and a camera, respectively, as the same object when the two objects satisfy predetermined conditions.

[0003] For example, Patent Document 2 discloses an object detection device that determines that objects detected by a radar sensor and a camera sensor are the same object, based on the condition that there is an overlapping area in the radar search area and the image search area.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2019-152617

[0007] Patent Document 2: Japanese Patent Application Publication No. 2017-194432 Summary of the invention

[0008] Problems to be solved by the invention

[0009] However, the techniques disclosed in Patent Documents 1 and 2 require projection processing of projecting the detection result of the camera onto the coordinate space of the radar or projecting the detection result of the radar onto the captured image of the camera. Therefore, the techniques disclosed in Patent Documents 1 and 2 have the following problems: the detection accuracy is degraded due to errors in the projection processing, or the processing time required to recognize the object is extended by an amount corresponding to the time for executing the projection processing.

[0010] Therefore, the present disclosure provides an object detection method and the like that can improve detection accuracy and easily reduce processing time.

[0011] Means for solving problems

[0012] The object detection method disclosed in the present invention obtains a first image and a second image, wherein each pixel of the second image corresponds one-to-one to each pixel of the first image; performs a first recognition process, in which the category of a first object appearing in the first image is recognized; performs a second recognition process, in which the position of a second object appearing in the second image is recognized; and performs a detection process to detect the first object and the second object as the same object when a first area based on the first object in the first image and a second area based on the second object in the second image overlap.

[0013] The object detection method disclosed in the present invention obtains an image; performs a first recognition process in which a category of a first object appearing in the image is recognized; performs a second recognition process in which a position of a second object appearing in the image is recognized; and performs a detection process to detect the first object and the second object as the same object when a first area based on the first object in the image overlaps with a second area based on the second object in the image.

[0014] The program according to the present disclosure causes one or more processors to execute the object detection method.

[0015] The object detection system disclosed in the present invention comprises: an acquisition unit, which acquires a first image and a second image, each pixel of the second image corresponds one-to-one to each pixel of the first image; a first recognition unit, which performs a first recognition process, in which the category of a first object appearing in the first image is recognized; a second recognition unit, which performs a second recognition process, in which the position of a second object appearing in the second image is recognized; and a detection unit, which performs a detection process to detect the first object and the second object as the same object when a first area based on the first object in the first image and a second area based on the second object in the second image overlap.

[0016] Effects of the Invention

[0017] According to an object detection method and the like according to one embodiment of the present disclosure, there are advantages in that detection accuracy can be improved and processing time can be easily reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a block diagram showing an overview of an object detection system according to an embodiment.

[0019] Figure 2 It is an explanatory diagram of the first area in the first image and the second area in the second image.

[0020] Figure 3 It is an explanatory diagram of the first scale and the second scale.

[0021] Figure 4 It is a diagram showing a specific example of detection processing by the object detection system according to the embodiment.

[0022] Figure 5 This is a flowchart showing an example of the operation of the object detection system according to the embodiment.

[0023] Figure 6It is a diagram showing a specific example of object detection by the object detection system according to the embodiment.

[0024] Figure 7 It is a diagram showing a specific example of the detection process of the object detection system according to the first modified example of the embodiment.

[0025] Figure 8 This is a block diagram showing an overview of an object detection system according to a second modified example of the embodiment. DETAILED DESCRIPTION

[0026] Hereinafter, the embodiments will be described in detail with reference to the drawings.

[0027] In addition, the embodiments described below are all general or specific examples. The numerical values, shapes, materials, components, configuration positions and connection methods of components, steps, and the order of steps shown in the following embodiments are examples and are not intended to limit the present disclosure.

[0028] <Structure>

[0029] Below, use Figure 1 The configuration of the object detection system according to the embodiment will be described. Figure 1 1 is a block diagram showing an overview of an object detection system 1 according to an embodiment. The object detection system 1 is a system for detecting an object shown in an image. In the embodiment, the object detection system 1 is mounted on a camera module 100 .

[0030] The camera module 100 includes a sensor unit 2 and an object detection system 1. The sensor unit 2 may be a component of the object detection system 1.

[0031] The camera module 100 includes a computer including a processor and a memory. The memory is a ROM (Read Only Memory) or a RAM (Random Access Memory), etc., and can store a program executed by the processor. The object detection system 1 is implemented by a processor etc. executing the program stored in the memory.

[0032] In an embodiment, the sensor unit 2 is a BW (Black and White)-TOF (Time Of Flight) sensor unit 2A having the functions of both a 2D camera and a 3D camera. The BW-TOF sensor unit 2A includes a BW-TOF sensor 21 and a light source 22. The BW-TOF sensor 21 is provided with pixels that receive near-infrared light (IR pixels) and pixels that receive visible light (BW pixels). The light source 22 emits near-infrared light. Moreover, the BW-TOF sensor unit 2A is capable of generating a brightness image (black and white image) through the BW pixels when the light source 22 does not emit light. In addition, when the light source 22 emits light, the BW-TOF sensor unit 2A is capable of receiving reflected light of the light irradiated by the light source 22 through the IR pixels, and generating a distance image based on the amount of light received. That is, the BW-TOF sensor unit 2A is capable of generating both a brightness image and a distance image.

[0033] The object detection system 1 includes an acquisition unit 11, a first recognition unit 12, a second recognition unit 13, a detection unit 14, and an output unit 15. In addition, the output unit 15 may not be a component of the object detection system 1.

[0034] The acquisition unit 11 acquires the first image I1 (see Figure 6 etc.) and the second image I2 (refer to Figure 6 The pixels of the second image I2 correspond one-to-one to the pixels of the first image I1. The first image I1 and the second image I2 are both two-dimensional images. Here, "the pixels of the first image I1 correspond one-to-one to the pixels of the second image I2" means that when the first image I1 and the second image I2 are overlapped, the pixels of the first image I1 and the pixels of the second image I2 overlap with almost no deviation.

[0035] The first image I1 and the second image I2 are both images generated based on the data captured by the BW-TOF sensor unit 2A. The first image I1 is a brightness image (black and white image) captured by BW pixels and representing each pixel with brightness. In addition, the second image I2 is a distance image of each pixel captured by IR pixels with distance. The "distance" mentioned here refers to the distance from the object shown in the second image I2 to the sensor unit 2 (or camera module 100).

[0036] In addition, the sensor having the functions of both a 2D camera and a 3D camera is not limited to the BW-TOF sensor unit 2A. For example, the sensor having the functions of both a 2D camera and a 3D camera may also be an RGB-TOF sensor. The RGB-TOF sensor is different from the BW-TOF sensor unit 2A in that it can generate a color image as a brightness image instead of a black-and-white image.

[0037] The first recognition unit 12 performs a recognition operation on the first object Ob1 (see FIG. 1 ) that appears in the first image I1. Figure 6 Here, the first recognition process is a process of recognizing the first object Ob1 shown in the first image I1 based on brightness.

[0038] Here, "identifying the first object Ob1 based on brightness" means identifying whether a predetermined target (object) is reflected in the first image I1 based on brightness. The target is the object to be identified, and is appropriately set according to the purpose of the object detection system 1. For example, in the case where the object detection system 1 is intended to detect a person, the target is a person. In this case, if a person is reflected in the first image I1, the person is identified as the first object Ob1 in the first identification process. On the other hand, if a tree is reflected in the first image I1, the tree is not identified as the first object Ob1 in the first identification process.

[0039] Furthermore, the target to be recognized in the first recognition process is not limited to a person, but may be another object. Furthermore, the target to be recognized in the first recognition process is not limited to one type, but may be multiple types.

[0040] In an embodiment, the first recognition unit 12 performs the first recognition process by using a learned model that is previously machine-learned to recognize a predetermined target when the first image I1 is input. For example, the first recognition unit 12 can perform the first recognition process by performing a pattern matching process on the first image I1 using the predetermined target as a pattern image.

[0041] The second recognition unit 13 performs a recognition operation on the second object Ob2 (refer to Figure 6 Here, the second recognition processing is a processing for recognizing the second object Ob2 shown in the second image I2 based on the distance.

[0042] Here, "recognizing the second object Ob2 based on the distance" means that a plurality of pixels whose distance (depth value) is close (i.e., the difference in depth value between pixels is smaller than the threshold value) are grouped as one and the group is recognized as a target. Therefore, in the second recognition process, unlike the first recognition process, the type of the target is irrelevant, so the second object Ob2 is not necessarily a predetermined target.

[0043] Figure 2 1 is an explanatory diagram of the first area A1 in the first image I1 and the second area A2 in the second image I2. Figure 2 In the example shown, it is assumed that the first image I1 and the second image I2 overlap. Figure 2In the example shown, it is assumed that the first object Ob1 and the second object Ob2 are the same object.

[0044] The first area A1 is an area based on the first object Ob1 in the first image I1. When the first object Ob1 is recognized in the first recognition process, the first area A1 is set in the first image I1. The second area A2 is an area based on the second object Ob2 in the second image I2. When the second object Ob2 is recognized in the second recognition process, the second area A2 is set in the second image I2.

[0045] In an embodiment, if Figure 2 As shown in FIG. 1 , the first area A1 is set in the first image I1 as a rectangular area including not only a plurality of pixels representing the first object Ob1 but also one or more pixels around the first object Ob1. Figure 2 As shown, the second area A2 is set in the second image I2 as a rectangular area including not only a plurality of pixels representing the second object Ob2 but also one or more pixels around the second object Ob2.

[0046] In addition, if Figure 2 As shown in FIG. 1 , the first area A1′ may also be set as an area including only a plurality of pixels representing the first object Ob1. Figure 2 As shown, the second area A2' may also be set as an area including only a plurality of pixels representing the second object Ob2.

[0047] When a first area A1 based on the first object Ob1 in the first image I1 and a second area A2 based on the second object Ob2 in the second image I2 overlap, the detection unit 14 performs a detection process to detect the first object Ob1 and the second object Ob2 as the same object. The detection process is a so-called fusion process. In an embodiment, a first determination process is performed in the detection process, in which a determination is made as to whether the first object Ob1 and the second object Ob2 are the same object based on whether the first ratio R_2d and the second ratio R_3d exceed a reference value. Here, the first ratio R_2d refers to an overlapping area A3 (refer to Figure 3 ) is the ratio of the overlapping region A3 to the first region A1. In addition, the second ratio R_3d refers to the ratio of the overlapping region A3 to the second region A2.

[0048] Figure 3 : is an illustration of the first ratio R_2d and the second ratio R_3d. Figure 3 As shown in (a), the area of ​​the first region A1 is represented by "A_2d", and the area of ​​the second region A2 is represented by "A_3d". Figure 3As shown in (b), "A_int" represents the area of ​​the repeated region A3. Figure 3 As shown in (c), the area of ​​the entire region including the first region A1 and the second region A2 is represented by "A_uni". In this case, the first ratio R_2d is represented by the formula "R_2d = A_int / A_2d". In addition, the second ratio R_3d is represented by the formula "R_3d = A_int / A_3d".

[0049] In the detection process, as described above, the first determination process is first performed. In the first determination process, the first object Ob1 and the second object Ob2 that satisfy the conditions that the first ratio R_2d is greater than the first threshold R_2dth as a reference value and the second ratio R_3d is greater than the second threshold R_3dth as a reference value are determined to be the same object. On the other hand, in the first determination process, the first object Ob1 and the second object Ob2 that do not satisfy the above conditions are determined to be not the same object. The first threshold R_2dth and the second threshold R_3dth can be appropriately set according to the detection accuracy required by the object detection system 1. The first threshold R_2dth is, for example, 0.05. In addition, the second threshold R_3dth is, for example, 0.3.

[0050] In the detection process, when there are multiple second objects Ob2 that are determined to be the same object as the first object Ob1 by the first determination process, a second determination process is performed to determine whether a second object Ob2 among the multiple second objects Ob2 is the same object as the first object Ob1 based on the size of the first ratio R_2d. In the second determination process, for example, the second object Ob2 having the highest first ratio R_2d among the multiple second objects Ob2 is determined to be the same object as the first object Ob1.

[0051] In the detection process, when there are multiple second objects Ob2 that are determined to be the same object as the first object Ob1 by the second determination process, a third determination process is performed to determine whether a second object Ob2 among the multiple second objects Ob2 is the same object as the first object Ob1 based on the distance. Here, the case where there are multiple second objects Ob2 that are determined to be the same object as the first object Ob1 in the second determination process corresponds to, for example, the case where there are multiple second objects Ob2 whose first ratio R_2d is substantially the same value. In the third determination process, for example, the second object Ob2 with the shortest distance among the multiple second objects Ob2 is determined to be the same object as the first object Ob1.

[0052] In addition, in the third determination process, it is also possible to determine whether a certain second object Ob2 among the plurality of second objects Ob2 is the same object as the first object Ob1 based on a parameter other than the distance. For example, in the third determination process, the second object Ob2 whose size of the second area A2 is closest to the size of the first area A1 may be determined to be the same object as the first object Ob1. In addition, for example, in the third determination process, the second object Ob2 whose representative value (for example, average value, maximum value or mode value, etc.) of the brightness values ​​of the plurality of pixels included in the second area A2 is the largest may be determined to be the same object as the first object Ob1. In addition, in the third determination process, it is also possible to determine whether a certain second object Ob2 among the plurality of second objects Ob2 is the same object as the first object Ob1 based on at least one of the above-mentioned distance, the sizes of the first area A1 and the second area A2, and the representative value of the brightness values ​​of the plurality of pixels included in the second area A2.

[0053] Below, use Figure 4 A specific example of the detection process will be described. Figure 4 FIG. 1 is a diagram showing a specific example of detection processing by the object detection system 1 according to the embodiment. Figure 4 In the example shown in (a), it is assumed that the first image I1 and the second image I2 overlap. Figure 4 In the example shown in (a), two first objects Ob11 and Ob12 are recognized in the first image I1 through the first recognition process. Figure 4 In the example shown in (a) of FIG. 1 , five second objects Ob21 , Ob22 , Ob23 , Ob24 , and Ob25 are recognized in the second image I2 through the second recognition processing.

[0054] In the detection process, it is determined whether the first object Ob11 and the five second objects Ob21, Ob22, Ob23, Ob24, and Ob25 are the same object. In addition, the second areas A23, A24, and A25 based on the three second objects Ob23, Ob24, and Ob25 do not overlap with the first area A11 based on the first object Ob11. Therefore, the description of the determination process of the three second objects Ob23, Ob24, and Ob25 is omitted below. Figure 4 As shown in (b) of FIG. 1 , only the determination process regarding the two second objects Ob21 and Ob22 is described.

[0055] First, in the detection process, the first determination process is performed on the combination of the first object Ob11 and the second object Ob21. If the first ratio in the combination is set to "R_2d_11" and the second ratio is set to "R_3d_11", then "R_2d_11 ≥ R_2dth" and "R_3d_11 < R_3dth", so the condition is not satisfied. Therefore, in the first determination process, it is determined that the first object Ob11 and the second object Ob21 are not the same object.

[0056] Next, in the detection process, the first determination process is performed on the combination of the first object Ob11 and the second object Ob22. If the first ratio in the combination is set to "R_2d_12" and the second ratio is set to "R_3d_12", then "R_2d_12 ≥ R_2dth" and "R_3d_12 ≥ R_3dth", and thus the condition is satisfied. Therefore, in the first determination process, it is determined that the first object Ob11 and the second object Ob22 are the same object. Thus, in the detection process, the first object Ob11 and the second object Ob22 are detected as the same object.

[0057] In addition, in the detection process, it is determined whether the first object Ob12 and the five second objects Ob21, Ob22, Ob23, Ob24, and Ob25 are the same object. In addition, the second areas A21 and A22 based on the two second objects Ob21 and Ob22 do not overlap with the first area A12 based on the first object Ob12. Therefore, the description of the determination process of the two second objects Ob21 and Ob22 is omitted below. Figure 4 As shown in (c), only the determination process regarding the three second objects Ob23, Ob24, and Ob25 is described.

[0058] First, in the detection process, the first determination process is performed on the combination of the first object Ob12 and the second object Ob23. If the first ratio in the combination is set to "R_2d_23" and the second ratio is set to "R_3d_23", then "R_2d_23<R_2dth" and "R_3d_23<R_3dth", so the condition is not satisfied. Therefore, in the first determination process, it is determined that the first object Ob12 and the second object Ob23 are not the same object.

[0059] Next, in the detection process, the first determination process is performed on the combination of the first object Ob12 and the second object Ob24. If the first ratio in the combination is set to "R_2d_24" and the second ratio is set to "R_3d_24", then "R_2d_24 ≥ R_2dth" and "R_3d_24 ≥ R_3dth", and thus the condition is satisfied. Therefore, in the first determination process, it is determined that the first object Ob12 and the second object Ob24 are the same object.

[0060] Next, in the detection process, the first determination process is performed on the combination of the first object Ob12 and the second object Ob25. If the first ratio in the combination is set to "R_2d_25" and the second ratio is set to "R_3d_25", then "R_2d_25 ≥ R_2dth" and "R_3d_25 ≥ R_3dth", and thus the condition is satisfied. Therefore, in the first determination process, it is determined that the first object Ob12 and the second object Ob25 are the same object.

[0061] Here, since there are two second objects Ob2 that are determined to be the same object as the first object Ob12 by the first determination process, the second determination process is performed in the detection process. Here, if the first ratio "R_2d_24" in the combination of the first object Ob12 and the second object Ob24 is compared with the first ratio "R_2d_25" in the combination of the first object Ob12 and the second object Ob25, it is "R_2d_24>R_2d_25". Therefore, in the second determination process, it is determined that the first object Ob12 and the second object Ob24 are the same object. Therefore, in the detection process, the first object Ob12 and the second object Ob24 are detected as the same object.

[0062] The output unit 15 performs an output process in which object information related to the object detected in the detection process of the detection unit 14 is output, including information indicating the category of the object recognized by the first recognition process and information indicating the position of the object recognized by the second recognition process. Specifically, the object information is information obtained by synthesizing the recognition result of the first object Ob1 and the recognition result of the second object Ob2 detected as the same object. The recognition result of the first object Ob1 is information indicating the category of the first object Ob1 recognized by the first recognition process. The recognition result of the second object Ob2 is information indicating the position (position in three-dimensional space) of the second object Ob2 recognized by the second recognition process.

[0063] For example, when the camera module 100 is mounted on a vehicle such as a car, the output unit 15 outputs the object information to an onboard ECU (Electronic Control Unit). In this case, the onboard ECU can use the object information for vehicle control. In addition, for example, the output unit 15 outputs the object information to an onboard display and displays it. In this case, the driver of the vehicle can use the object information for driving the vehicle by observing the onboard display.

[0064] In addition, in the output processing of the output unit 15, the information indicating the moving speed of the object represented by the object information (i.e., the object detected in the detection processing) or the moving direction of the object may also be included in the object information and output. The moving speed of the object can be calculated by referring to the time series data of the information indicating the position of the second object Ob2. Similarly, the moving direction of the object can be calculated by referring to the time series data of the information indicating the position of the second object Ob2.

[0065] In addition, the object detection system 1 may also perform a tracking process for an object represented by the tracking object information (i.e., an object detected in the detection process). In the tracking process, the time series data of the object is obtained by tracking the object detected in the detection process. Furthermore, in the output process of the output unit 15, the result of the tracking process may be output.

[0066] For example, if the output unit 15 outputs the result of the tracking process to the vehicle-mounted ECU, the vehicle-mounted ECU can control the vehicle based on the detected trajectory of the object. In addition, for example, if the output unit 15 outputs the result of the tracking process to the vehicle-mounted display and displays it, the driver of the vehicle can drive the vehicle based on the detected trajectory of the object by observing the vehicle-mounted display.

[0067] <Action>

[0068] Below, use Figure 5 An operation example of the object detection system 1 according to the embodiment will be described.

[0069] First, the acquisition unit 11 of the object detection system 1 acquires the first image I1 and the second image I2 (step S1 ). In the embodiment, the acquisition unit 11 acquires the brightness image and the distance image captured by the BW-TOF sensor unit 2A as the sensor unit 2 to acquire the first image I1 and the second image I2.

[0070] Next, the first recognition unit 12 of the object detection system 1 performs a first recognition process of recognizing the category of the first object Ob1 in the first image I1 (step S2). In addition, the second recognition unit 13 of the object detection system 1 performs a second recognition process of recognizing the position of the second object Ob2 in the second image I2 (step S3). In addition, step S2 and step S3 may be performed in reverse order instead of in this order. In addition, step S2 and step S3 may be performed in parallel.

[0071] Next, the detection unit 14 of the object detection system 1 performs a detection process. Specifically, it is determined whether the first object Ob1 identified in the first recognition process and the second object Ob2 identified in the second recognition process are the same object (step S4). In the embodiment, the detection unit 14 first performs a first determination process. Furthermore, in the case where there are multiple second objects Ob2 that are determined to be the same object as the first object Ob1 by the first determination process, the detection unit 14 also performs a second determination process. Furthermore, in the case where there are multiple second objects Ob2 that are determined to be the same object as the first object Ob1 by the second determination process, the detection unit 14 also performs a third determination process.

[0072] If the first object Ob1 and the second object Ob2 are detected as the same object by the detection process, the object detection system 1 combines the recognition result of the first object Ob1 and the recognition result of the second object Ob2 determined to be the same object (step S5). Step S5 can be executed by the detection unit 14 or the output unit 15.

[0073] Furthermore, the object detection system 1 executes tracking processing (step S6) as necessary. Step S6 may be executed by the detection unit 14 or by the output unit 15. Whether or not to execute the tracking processing may be set appropriately.

[0074] Then, the output unit 15 of the object detection system 1 performs output processing (step S7). In the embodiment, the output unit 15 outputs object information including at least information indicating the type of the object detected by the detection processing and information indicating the position of the object.

[0075] Figure 6 FIG. 1 is a diagram showing a specific example of object detection by the object detection system 1 according to the embodiment. Figure 6 In the example shown, it is assumed that the first recognition process performed by the first recognition unit 12 recognizes a person as a predetermined target.

[0076] like Figure 6As shown, in the first recognition process, the first object Ob11 of the human shape shown in the first image I1 is recognized, and the tree is not recognized as the first object Ob1. Therefore, the first area A11 based on the first object Ob11 is set in the first image I1. In addition, in the second recognition process, the second object Ob21 of the human shape and the second object Ob22 of the tree shape shown in the second image I2 are recognized. Therefore, the second area A21 based on the second object Ob21 of the human shape and the second area A22 based on the second object Ob22 of the tree shape are set in the second image I2.

[0077] And, in Figure 6 In the example shown, since the first area A11 and the second area A21 overlap, the object detection system 1 detects the first human-shaped object Ob11 and the second human-shaped object Ob21 as the same object. Therefore, the object detection system 1 outputs object information with the object type being "human" and the object position being "(x1, y1, z1)". "(x1, y1, z1)" is the coordinate indicating the position of the second object Ob21 recognized by the second recognition process.

[0078] In addition, Figure 6 In the example shown, although there is no first area A1 that overlaps with the second area A22, the object detection system 1 also outputs object information about the tree-shaped second object Ob22. In this object information, the position of the object is set to "(x2, y2, z2)" and the type of the object is set to "unknown". "(x2, y2, z2)" is the coordinate indicating the position of the second object Ob22 recognized by the second recognition processing. In this way, the object detection system 1 can also output not only information related to the objects detected as the same object by the detection processing, but also information related to the objects that are not detected as the same object.

[0079] <Advantages>

[0080] The advantages of the object detection system 1 of the embodiment are described below. In the techniques disclosed in Patent Documents 1 and 2, before performing the so-called fusion processing, it is necessary to perform a projection processing of projecting the detection result of the camera into the coordinate space of the radar, or projecting the detection result of the radar into the captured image of the camera. Therefore, in the techniques disclosed in Patent Documents 1 and 2, there are problems such as the detection accuracy deteriorating due to the occurrence of errors in the projection processing, or the processing time required to recognize the object being extended by an amount corresponding to the time of performing the projection processing.

[0081] In contrast, in the object detection system 1 of the embodiment, the first recognition process is performed on the first image I1, and the second recognition process is performed on the second image I2 whose pixels correspond one-to-one to the pixels of the first image I1, and then the detection process (fusion process) is performed. Therefore, in the object detection system 1 of the embodiment, the first area A1 and the second area A2 can be set in the first image I1 and the second image I2, respectively, so the projection process as described above is not required. Therefore, in the object detection system 1 of the embodiment, there is no problem associated with performing the projection process, so there is an advantage that the detection accuracy can be improved and the processing time can be easily reduced.

[0082] (Other embodiments)

[0083] The object detection system of one or more embodiments of the present disclosure is described above based on the embodiments, but the present disclosure is not limited to these embodiments. As long as it does not deviate from the main purpose of the present disclosure, the various deformations that a person skilled in the art can think of in each embodiment and the methods constructed by combining the constituent elements in different embodiments can also be included in the scope of one or more embodiments of the present disclosure.

[0084] <First Modification>

[0085] Figure 7 1 is a diagram showing a specific example of the detection process of the object detection system 1 according to the first variant of the embodiment. The detection process performed by the object detection system 1 according to the first variant is different from the detection process performed by the object detection system 1 according to the embodiment. In addition, the description of the points common to the object detection system 1 according to the embodiment is omitted below.

[0086] like Figure 7 As shown in the example (a) of FIG. 1 , when an object having a complex shape such as an agricultural machine is shown in the second image I2, the object may not be recognized as one second object Ob2 but as multiple second objects Ob2 in the second recognition process. Figure 7 In the example shown in (a), in the first recognition process, the agricultural machine type object is recognized as one first object Ob11. On the other hand, in the second recognition process, the agricultural machine type object is not recognized as one second object Ob2, but is recognized as four second objects Ob21, Ob22, Ob23, and Ob24.

[0087] In this case, the second areas A21, A22, A23, A24 based on the second objects Ob21, Ob22, Ob23, Ob24 are smaller than the second area A2 when the agricultural machine-type object is used as one second object Ob2. Therefore, there is a problem that the first ratio R_2d of each of the second objects Ob21, Ob22, Ob23, Ob24 becomes smaller, and it is difficult to determine that they are the same object as the first object Ob11 in the first determination process.

[0088] Therefore, in the first modified example, in the detection process, when there are multiple second objects Ob2 identified by the second recognition process, and the second area A2 based on each of the multiple second objects Ob2 overlaps with the first area A1 based on the same first object Ob1, and the distances between the multiple second objects Ob2 are within a predetermined range, the multiple second objects Ob2 are synthesized. In addition, in the detection process, it is determined whether the synthesized second object Ob2 and the first object Ob1 are the same object.

[0089] First, in the detection process, in each combination of the first object Ob11 and the plurality of second objects Ob2, two or more second objects Ob2 whose second ratio R_3d is greater than or equal to the second threshold value R_3dth are determined. Figure 7 In the example shown in (a), the four second objects Ob21, Ob22, Ob23, and Ob24 correspond to the two or more second objects Ob2. Figure 7 In the example shown in (a), the second areas A21, A22, A23, A24 based on the four second objects Ob21, Ob22, Ob23, Ob24 overlap with the first area A11 based on the same first object Ob11. Furthermore, the distances of the plurality of second objects Ob21, Ob22, Ob23, Ob24 all converge within a predetermined range.

[0090] Therefore, in the detection process, for example Figure 7 As shown in (b), by synthesizing the four second objects Ob21, Ob22, Ob23, and Ob24, a second area A21' based on a second object Ob21' including the four second objects Ob21, Ob22, Ob23, and Ob24 is set in the second image I2. Figure 7 In the example shown in (b), the second area A21' is a rectangular area including the second areas A21, A22, A23, and A24, but the present invention is not limited thereto. For example, the second area A21' may be an area including only the second areas A21, A22, A23, and A24.

[0091] After that, the detection unit 14 performs the first determination process on the combination of the first object Ob11 and the synthesized second object Ob21'. Thus, the detection unit 14 can detect the first object Ob11 and the synthesized second object Ob21' as the same object. In addition, when there are multiple synthesized second objects Ob2 that are determined to be the same object as the first object Ob1 by the first determination process, the detection unit 14 also performs the second determination process. And, when there are multiple second objects Ob2 that are determined to be the same object as the first object Ob1 by the second determination process, the detection unit 14 also performs the third determination process.

[0092] Here, in the detection process, the position of a second object Ob2 among the multiple second objects Ob2 to be synthesized may be calculated as the position of the detected object. For example, the position of the second object Ob2 with the shortest distance among the multiple second objects Ob2 may be calculated as the position of the detected object. In addition, for example, the position of the second object Ob2 with the largest size of the second area A2 among the multiple second objects Ob2 may be calculated as the position of the detected object. In addition, for example, the position of the second object Ob2 with the largest number of pixels recognized as the second object Ob2 among the multiple second objects Ob2 may be calculated as the position of the detected object. In addition, for example, the position of the second object Ob2 with the highest brightness of the pixels recognized as the second object Ob2 among the multiple second objects Ob2 may be calculated as the position of the detected object.

[0093] In this case, the position of the detected object can be calculated as any one of the coordinates of the closest point to the sensor unit 2, the far coordinates of the farthest point from the sensor unit 2, and the center of gravity coordinates of the second object Ob2 selected as described above.

[0094] In addition, during the detection process, the position of the synthesized second object Ob2 may be calculated as the position of the detected object.

[0095] In this case, the position of the detected object can be calculated as any one of the coordinates of the closest point from the sensor unit 2, the far coordinates of the farthest point from the sensor unit 2, and the center of gravity coordinates of the synthesized second object Ob2.

[0096] <Second Modification>

[0097] Figure 8 1 is a block diagram showing an overview of an object detection system 1A according to a second modification of the embodiment. The object detection system 1A according to the second modification differs from the object detection system 1 according to the embodiment in that the sensor unit 2 is not a BW-TOF sensor unit 2A but a TOF sensor unit 2B. In addition, the following description of the points common to the object detection system 1 according to the embodiment will be omitted.

[0098] The TOF sensor unit 2B is a distance sensor. The TOF sensor unit 2B includes a TOF sensor 21B and a light source 22. The TOF sensor 21B is configured with pixels (IR pixels) that receive near-infrared light, and can generate a brightness image (IR brightness image) and a distance image of near-infrared light. Therefore, the acquisition unit 11 of the object detection system 1A of the second modified example acquires a brightness image (IR brightness image) in which each pixel is represented by brightness as a first image I1, and acquires a distance image in which each pixel is represented by distance as a second image I2.

[0099] <Other Modifications>

[0100] For example, in the embodiment, in the first determination process, it is also possible to determine whether the first object Ob1 and the second object Ob2 are the same object based on the third ratio IoU, which is the ratio of the overlapping area A3 to the entire area including the first area A1 and the second area A2. The third ratio IoU is represented by the mathematical formula "IoU = A_int / A_uni". For example, in the first determination process, it is also possible to determine that the first object Ob1 and the second object Ob2 that meet the condition that the third ratio IoU is greater than the third threshold value IoU_th as a reference value are the same object, and determine that the first object Ob1 and the second object Ob2 that do not meet the above condition are not the same object.

[0101] In addition, in the first determination process, the first ratio R_2d, the second ratio R_3d, and the third ratio IoU may be combined to determine whether the first object Ob1 and the second object Ob2 are the same object. That is, in the detection process, it is also possible to determine whether the first object Ob1 and the second object Ob2 are the same object based on whether at least one of the first ratio R_2d, the second ratio R_3d, and the third ratio IoU exceeds the reference value.

[0102] For example, in an embodiment, when there are multiple second objects Ob2 determined to be the same object as the first object Ob1 in the first determination process, in the second determination process, it is also possible to determine whether a particular second object Ob2 among the multiple second objects Ob2 is the same object as the first object Ob1 based on the size of at least one of the first ratio R_2d, the second ratio R_3d, and the third ratio IoU.

[0103] For example, in the embodiment, the first image I1 and the second image I2 may also be one image. For example, when the sensor unit 2 is a TOF sensor, the distance image output by the TOF sensor may be used as the first image I1 and the second image I2. In other words, the distance image is the first image I1 and the second image I2. In this case, the first recognition process and the second recognition process may be performed on the distance image, respectively.

[0104] For example, in the embodiment, the object detection system 1 is mounted on the camera module 100 , but the present invention is not limited thereto. For example, the object detection system 1 may be implemented by a device other than the camera module 100 , such as a personal computer.

[0105] For example, the present disclosure can be implemented not only as an object detection system but also as an object detection method including steps (processing) performed by components constituting the object detection system.

[0106] An object detection method is a method performed by an object detection system. Figure 5 As shown, a first image I1 and a second image I2 in which each pixel corresponds one-to-one with each pixel of the first image I1 are obtained (step S1), a first recognition process is performed to recognize the category of the first object Ob1 reflected in the first image I1 (step S2), a second recognition process is performed to recognize the position of the second object Ob2 reflected in the second image I2 (step S3), and when a first area A1 based on the first object Ob1 in the first image I1 and a second area A2 based on the second object Ob2 in the second image I2 overlap, a detection process is performed to detect the first object Ob1 and the second object Ob2 as the same object (step S4).

[0107] In addition, in the object detection method, when the first image I1 and the second image I2 are one image as described above, one image is obtained, a first recognition process is performed to recognize the category of the first object Ob1 reflected in the one image, a second recognition process is performed to recognize the position of the second object Ob2 reflected in the one image, and when a first area A1 based on the first object Ob1 in one image and a second area A2 based on the second object Ob2 in one image overlap, a detection process is performed to detect the first object Ob1 and the second object Ob2 as the same object.

[0108] For example, the present disclosure can be implemented as a program for causing a computer (processor) to execute the steps included in the object detection method. The processor can be one or more. Furthermore, the present disclosure can be implemented as a non-temporary computer-readable recording medium such as a CD-ROM having the program recorded thereon.

[0109] For example, when the present disclosure is implemented by a program (software), each step is performed by executing the program using hardware resources such as a CPU, a memory, and an input / output circuit of a computer. That is, the CPU obtains data from the memory or the input / output circuit and performs calculations, or outputs calculation results to the memory or the input / output circuit, thereby performing each step.

[0110] In addition, in the above-mentioned embodiment, each component included in the object detection system may be formed by dedicated hardware, or implemented by executing a software program suitable for each component. Each component may also be implemented by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory.

[0111] Part or all of the functions of the object detection system of the above-mentioned embodiment are typically implemented as an integrated circuit, i.e., LSI. They can be either individually single-chip or in a single-chip manner including part or all. In addition, integrated circuitization is not limited to LSI, and can also be implemented by a dedicated circuit or a general-purpose processor. It is also possible to use an FPGA (Field Programmable Gate Array) that can be programmed after LSI manufacturing, or a reconfigurable processor that can reconfigure the connection and setting of the circuit unit inside the LSI.

[0112] (Note)

[0113] The following techniques are disclosed through the description of the above embodiments.

[0114] (Technique 1) An object detection method, wherein a first image and a second image are obtained, each pixel of the second image corresponds one-to-one to each pixel of the first image; a first recognition process is performed, in which the category of a first object appearing in the first image is recognized; a second recognition process is performed, in which the position of a second object appearing in the second image is recognized; and in a case where a first area based on the first object in the first image overlaps with a second area based on the second object in the second image, a detection process is performed to detect the first object and the second object as the same object.

[0115] Thus, the first area and the second area can be set in the first image and the second image, respectively, so there is no need to perform projection processing of projecting the detection result of the camera into the coordinate space of the radar, or projecting the detection result of the radar into the image captured by the camera. Therefore, there is no problem associated with performing the projection processing, so there is an advantage of achieving improved detection accuracy and easily reducing processing time.

[0116] (Technique 2) The object detection method described in Technique 1, wherein the first image and the second image are both images generated based on data captured by one sensor unit.

[0117] This has the advantage that each pixel of the first image and each pixel of the second image can be easily matched one-to-one with each pixel of the second image, compared with the case where the first image and the second image are acquired from data captured by a plurality of sensor units.

[0118] (Technology 3) The object detection method described in Technology 2, wherein the sensor unit is a sensor composed of pixels that receive near-infrared light, the first image is a brightness image in which each pixel is represented by the brightness of the near-infrared light, and the second image is a distance image in which each pixel is represented by a distance calculated based on the amount of light received by the near-infrared light.

[0119] This has the advantage of being able to acquire the first image and the second image without using a 2D camera.

[0120] (Technique 4) The object detection method described in Technique 2, wherein the sensor unit is a sensor in which pixels receiving visible light and pixels receiving near-infrared light are integrally arranged, the first image is a brightness image captured by pixels receiving the visible light and each pixel is represented by the brightness of the visible light, and the second image is a distance image captured by pixels receiving the near-infrared light and each pixel is represented by a distance calculated based on the amount of light received by the near-infrared light.

[0121] Thus, since the 2D camera and the 3D camera are integrally provided, there is an advantage that the sensor unit can be easily miniaturized.

[0122] (Technique 5) An object detection method, wherein an image is obtained, a first recognition process is performed to recognize the category of a first object reflected in the image, a second recognition process is performed to recognize the position of a second object reflected in the image, and when a first area in the image based on the first object and a second area in the image based on the second object overlap, a detection process is performed to detect the first object and the second object as the same object.

[0123] Thus, the first area and the second area can be set in one image, so there is no need to perform projection processing of projecting the detection result of the camera into the coordinate space of the radar, or projecting the detection result of the radar into the image captured by the camera. Therefore, there is no problem associated with performing the projection processing, so there is an advantage of achieving improved detection accuracy and easily reducing processing time.

[0124] (Technique 6) An object detection method as described in any one of Techniques 1 to 5, wherein, in the detection process, a first determination process is performed, in which whether the first object and the second object are the same object is determined based on whether at least one of a first proportion of an overlapping area of ​​the first area and the second area in the first area, a second proportion of the overlapping area in the second area, and a third proportion of the overlapping area in the entire area including the first area and the second area exceeds a reference value.

[0125] Thus, the first object and the second object satisfying the condition that at least one of the first ratio, the second ratio, and the third ratio exceeds the reference value are detected as the same object, which has the advantage of easily improving the detection accuracy.

[0126] (Technique 7) The object detection method described in Technique 6, wherein, in the detection process, when there are multiple second objects that are determined to be the same object as the first object by the first determination process, a second determination process is performed, and in the second determination process, whether a second object among the multiple second objects is the same object as the first object is determined based on the size of at least one of the first ratio, the second ratio, and the third ratio.

[0127] Therefore, when there are multiple second objects determined to be the same object as the first object, one of the second objects can be selected based on the size of at least one of the first ratio, the second ratio and the third ratio, which has the advantage of easily improving detection accuracy.

[0128] (Technique 8) The object detection method described in Technique 7, wherein, in the detection process, when there are multiple second objects determined to be the same object as the first object by the second determination process, a third determination process is performed, and in the third determination process, it is determined whether a second object among the multiple second objects is the same object as the first object based on at least one of the distance, the sizes of the first area and the second area, and the representative value of the brightness values ​​of the multiple pixels contained in the second area.

[0129] Therefore, when there are multiple second objects that are determined to be the same object as the first object, one of the second objects can be selected based on the distance, the sizes of the first area and the second area, and at least one of the representative values ​​of the brightness values ​​of multiple pixels contained in the second area, thereby having the advantage of easily improving detection accuracy.

[0130] (Technique 9) An object detection method as described in any one of Techniques 1 to 8, wherein, in the detection process, when there are multiple second objects identified by the second recognition process, and the second areas based on each of the multiple second objects overlap with the first area based on the same first object, and the distance between the multiple second objects converges within a specified range, the multiple second objects are synthesized to determine whether the synthesized second object is the same object as the first object.

[0131] Thus, even when an actually single object is recognized as a plurality of second objects, the plurality of second objects can be detected as a single object, which has the advantage of easily improving detection accuracy.

[0132] (Technique 10) The object detection method described in Technique 9, wherein, in the detection process, the position of any one of the plurality of second objects to be synthesized is calculated as the position of the detected object.

[0133] This provides an advantage in that the accuracy of calculating the position of the detected object can be easily improved because the position of any one of the plurality of second objects to be synthesized is referenced.

[0134] (Technique 11) The object detection method described in Technique 9, wherein, in the detection process, the position of the synthesized second object is calculated as the position of the detected object.

[0135] This has the advantage that the accuracy of calculating the position of the detected object can be easily improved because the position of the synthesized second object is referenced.

[0136] (Technique 12) An object detection method as described in any one of Techniques 1 to 11, wherein output processing is also performed, in which object information related to the object detected by the detection processing is output including information indicating the category of the object identified by the first recognition processing and information indicating the position of the object identified by the second recognition processing.

[0137] This allows the type and position of the object detected by the detection process to be referenced in the system or the like at the output destination, thereby providing an advantage in that control based on the type and position of the object can be easily executed.

[0138] (Technique 13) The object detection method described in Technique 12, wherein, in the output processing, information indicating the moving speed of the object represented by the object information or the moving direction of the object is also included in the object information for output.

[0139] This allows reference to the moving speed or moving direction of the object detected by the detection process in the system or the like at the output destination, thereby providing an advantage of easily executing control based on the moving speed or moving direction of the object.

[0140] (Technique 14) The object detection method described in Technique 12 or 13, wherein a tracking process for tracking the object represented by the object information is also performed, and in the output process, a result of the tracking process is also output.

[0141] This allows reference to the trajectory of the object detected by the detection process in the system or the like at the output destination, and therefore provides an advantage in that control based on the trajectory of the object can be easily performed.

[0142] (Technique 15) A program that causes one or more processors to execute the object detection method according to any one of Techniques 1 to 14.

[0143] This makes it possible to provide a program that can improve detection accuracy and easily reduce processing time.

[0144] (Technique 16) An object detection system, comprising: an acquisition unit that acquires a first image and a second image in which each pixel corresponds one-to-one to each pixel of the first image; a first recognition unit that performs a first recognition process in which the category of a first object appearing in the first image is recognized; a second recognition unit that performs a second recognition process in which the position of a second object appearing in the second image is recognized; and a detection unit that performs a detection process in which, when a first area based on the first object in the first image overlaps with a second area based on the second object in the second image, detects the first object and the second object as the same object.

[0145] This makes it possible to provide an object detection system that can improve detection accuracy and easily reduce processing time.

[0146] Industrial Applicability

[0147] The present disclosure can be applied to a device or the like that recognizes an object by so-called fusion processing.

[0148] Description of Reference Numerals

[0149] 1.1A Object Detection System

[0150] 11 Acquisition Department

[0151] 12 First identification section

[0152] 13 Second identification part

[0153] 14. Inspection Department

[0154] 15 Output

[0155] 2 Sensor unit

[0156] 2A BW-TOF sensor unit

[0157] 21BW-TOF sensor

[0158] 22 Light Source

[0159] 2B TOF sensor unit

[0160] 21B TOF sensor

[0161] 100 Camera Module

[0162] A1, A1', A11, A12 first area

[0163] A2, A2', A21, A22, A23, A24, A25 Second Area

[0164] The second region after A21' synthesis

[0165] A3 Repeat Region

[0166] A_2D The area of ​​the first region

[0167] A_3D The area of ​​the second region

[0168] A_intThe area of ​​the repeated region

[0169] The area of ​​the entire A_uni region

[0170] I1 First image

[0171] I2 Second image

[0172] Ob1, Ob11, Ob12 first object

[0173] Ob2, Ob21, Ob22, Ob23, Ob24, Ob25 second object

[0174] Ob21' The second object after synthesis

Claims

1. A method for object detection, wherein: A first image and a second image are obtained, wherein each pixel of the second image corresponds one-to-one to each pixel of the first image, performing a first recognition process in which a type of a first object appearing in the first image is recognized, performing a second recognition process in which a position of a second object reflected in the second image is recognized, When a first area based on the first object in the first image overlaps with a second area based on the second object in the second image, a detection process is performed to detect the first object and the second object as the same object.

2. The object detection method according to claim 1, wherein: The first image and the second image are both images generated based on data photographed by one sensor unit.

3. The object detection method according to claim 2, wherein: The sensor unit is a sensor composed of pixels that receive near-infrared light. The first image is a brightness image in which each pixel is represented by the brightness of the near-infrared light. The second image is a distance image in which each pixel is represented by a distance calculated based on the amount of received near-infrared light.

4. The object detection method according to claim 2, wherein: The sensor unit is a sensor in which pixels for receiving visible light and pixels for receiving near-infrared light are integrally provided. The first image is a brightness image captured by pixels receiving the visible light and each pixel is represented by the brightness of the visible light. The second image is a distance image in which pixels that receive the near-infrared light are captured and each pixel is represented by a distance calculated based on the amount of received near-infrared light.

5. A method for object detection, wherein: Get an image, performing a first recognition process in which a type of a first object appearing in the one image is recognized, performing a second recognition process in which a position of a second object appearing in the one image is recognized, When a first region based on the first object in the one image overlaps with a second region based on the second object in the one image, a detection process is performed to detect the first object and the second object as the same object.

6. The object detection method according to any one of claims 1 to 5, wherein: In the detection process, a first determination process is performed, in which whether the first object and the second object are the same object is determined based on whether at least one of a first proportion of a repeated area where the first area overlaps with the second area in the first area, a second proportion of the repeated area in the second area, and a third proportion of a repeated area in the entire area including the first area and the second area exceeds a reference value.

7. The object detection method according to claim 6, wherein: In the detection process, when there are multiple second objects determined to be the same object as the first object by the first determination process, a second determination process is performed. In the second determination process, whether a second object among the multiple second objects is the same object as the first object is determined based on the size of at least one of the first ratio, the second ratio, and the third ratio.

8. The object detection method according to claim 7, wherein: In the detection process, when there are multiple second objects determined to be the same object as the first object by the second determination process, a third determination process is performed. In the third determination process, whether a second object among the multiple second objects is the same object as the first object is determined based on the distance, the sizes of the first area and the second area, and at least one of the representative values ​​of the brightness values ​​of multiple pixels included in the second area.

9. The object detection method according to any one of claims 1 to 8, wherein: In the detection process, when there are multiple second objects identified by the second recognition process, and the second areas based on each of the multiple second objects overlap with the first area based on the same first object, and the distance between the multiple second objects converges within a specified range, the multiple second objects are synthesized to determine whether the synthesized second objects are the same object as the first object.

10. The object detection method according to claim 9, wherein: In the detection process, the position of any one of the plurality of second objects to be synthesized is calculated as the position of the detected object.

11. The object detection method according to claim 9, wherein: In the detection process, the position of the synthesized second object is calculated as the position of the detected object.

12. The object detection method according to any one of claims 1 to 11, wherein: Output processing is also performed, in which object information related to the object detected by the detection processing is output including information indicating the category of the object identified by the first recognition processing and information indicating the position of the object identified by the second recognition processing.

13. The object detection method according to claim 12, wherein: In the output process, information indicating the moving speed of the object indicated by the object information or the moving direction of the object is further included in the object information and outputted.

14. The object detection method according to claim 12 or 13, wherein: further performing a tracking process of tracking the object represented by the object information, In the output process, the result of the tracking process is also output. 15 . A program causing one or more processors to execute the object detection method according to claim 1 .

16. An object detection system, wherein: have: An acquisition unit, which acquires a first image and a second image, wherein each pixel of the second image corresponds to each pixel of the first image in a one-to-one manner; A first recognition unit that performs a first recognition process in which a type of a first object that appears in the first image is recognized; A second recognition unit that performs a second recognition process in which a position of a second object that appears in the second image is recognized; as well as The detection unit performs a detection process for detecting the first object and the second object as the same object when a first area based on the first object in the first image overlaps with a second area based on the second object in the second image.

Images