Object detection method, program, and object detection system

Abstract

Provided is an object detection method including: acquiring a first image and a second image in which each pixel corresponds one-to-one to each pixel of the first image (step S1); executing a first recognition process for recognizing the type of a first object captured in the first image (step S2); executing a second recognition process for recognizing the position of a second object captured in the second image (step S3), and when a first region based on the first object in the first image and a second region based on the second object in the second image overlap, executing a detection process for detecting the first object and the second object as the same object (step S4).

Description

Object detection method, program, and object detection system

[0001] The present disclosure relates to an object detection method, a program, and an object detection system.

[0002] For example, Patent Document 1 discloses an object recognition device that recognizes two objects detected using a radar and a camera as the same object if the two objects satisfy predetermined conditions.

[0003] Furthermore, for example, Patent Document 2 discloses an object detection device that determines that an object detected by a radar sensor and a camera sensor is the same object, provided that there is an overlapping area between the radar search area and the image search area.

[0004] JP 2019-152617 A JP 2017-194432 A

[0005] However, the techniques disclosed in Patent Documents 1 and 2 require a projection process in which the detection results of the camera are projected onto the coordinate space of the radar, or the detection results of the radar are projected onto an image captured by the camera. Therefore, the techniques disclosed in Patent Documents 1 and 2 have problems such as a deterioration in detection accuracy due to errors occurring in the projection process, and an increase in the processing time required to recognize an object due to the time required to execute the projection process.

[0006] Therefore, the present disclosure provides an object detection method and the like that can easily reduce processing time while improving detection accuracy.

[0007] The object detection method according to the present disclosure acquires a first image and a second image in which each pixel corresponds one-to-one to each pixel of the first image, performs a first recognition process to recognize the type of a first object appearing in the first image, performs a second recognition process to recognize the position of a second object appearing in the second image, and performs a detection process to detect the first object and the second object as the same object when a first area based on the first object in the first image and a second area based on the second object in the second image overlap.

[0008] The object detection method according to the present disclosure acquires an image, performs a first recognition process to recognize the type of a first object appearing in the image, performs a second recognition process to recognize the position of a second object appearing in the image, and performs a detection process to detect the first object and the second object as the same object when a first area based on the first object in the image and a second area based on the second object in the image overlap.

[0009] The program according to the present disclosure causes one or more processors to execute the object detection method.

[0010] The object detection system according to the present disclosure includes an acquisition unit that acquires a first image and a second image in which each pixel corresponds one-to-one to each pixel of the first image; a first recognition unit that performs a first recognition process that recognizes the type of a first object appearing in the first image; a second recognition unit that performs a second recognition process that recognizes the position of a second object appearing in the second image; and a detection unit that performs a detection process that detects the first object and the second object as the same object when a first area based on the first object in the first image and a second area based on the second object in the second image overlap.

[0011] The object detection method according to one aspect of the present disclosure has the advantage of easily reducing processing time while improving detection accuracy.

[0012] FIG. 1 is a block diagram showing an overview of an object detection system according to an embodiment. FIG. 2 is an explanatory diagram of a first region in a first image and a second region in a second image. FIG. 3 is an explanatory diagram of a first ratio and a second ratio. FIG. 4 is a diagram showing a specific example of detection processing by the object detection system according to an embodiment. FIG. 5 is a flowchart showing an example of operation of the object detection system according to an embodiment. FIG. 6 is a diagram showing a specific example of object detection by the object detection system according to an embodiment. FIG. 7 is a diagram showing a specific example of detection processing by the object detection system according to a first modified example of the embodiment. FIG. 8 is a block diagram showing an overview of an object detection system according to a second modified example of the embodiment.

[0013] Hereinafter, the embodiments will be specifically described with reference to the drawings.

[0014] The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, the arrangement and connection of the components, steps, and the order of steps shown in the following embodiments are merely examples and are not intended to limit the present disclosure.

[0015] <Configuration> The configuration of an object detection system according to an embodiment will be described below with reference to Fig. 1. Fig. 1 is a block diagram showing an overview of an object detection system 1 according to an embodiment. The object detection system 1 is a system for detecting an object appearing in an image. In the embodiment, the object detection system 1 is mounted on a camera module 100.

[0016] The camera module 100 includes a sensor unit 2 and an object detection system 1. The sensor unit 2 may be a component of the object detection system 1.

[0017] The camera module 100 includes a computer including a processor, a memory, etc. The memory is a read-only memory (ROM) or a random access memory (RAM), etc., and can store a program to be executed by the processor. The object detection system 1 is realized by the processor, etc., that executes the program stored in the memory.

[0018] In this embodiment, the sensor unit 2 is a BW (Black and White)-TOF (Time of Flight) sensor unit 2A that combines the functions of a 2D camera and a 3D camera. The BW-TOF sensor unit 2A includes a BW-TOF sensor 21 and a light source 22. The BW-TOF sensor 21 includes pixels that receive near-infrared light (IR pixels) and pixels that receive visible light (BW pixels). The light source 22 emits near-infrared light. When the light source 22 is not emitting light, the BW-TOF sensor unit 2A can generate a luminance image (black and white image) using the BW pixels. When the light source 22 is emitting light, the BW-TOF sensor unit 2A can receive reflected light from the light source 22 using the IR pixels and generate a distance image based on the amount of light received. That is, the BW-TOF sensor unit 2A can generate both a brightness image and a range image.

[0019] The object detection system 1 includes an acquisition unit 11, a first recognition unit 12, a second recognition unit 13, a detection unit 14, and an output unit 15. Note that the output unit 15 does not necessarily have to be a component of the object detection system 1.

[0020] The acquisition unit 11 acquires a first image I1 (see FIG. 6, etc.) and a second image I2 (see FIG. 6, etc.) in which each pixel corresponds one-to-one to each pixel of the first image I1. Both the first image I1 and the second image I2 are two-dimensional images. Here, "each pixel of the first image I1 corresponds one-to-one to each pixel of the second image I2" means that when the first image I1 and the second image I2 are superimposed, each pixel of the first image I1 and each pixel of the second image I2 overlap with almost no misalignment.

[0021] The first image I1 and the second image I2 are both images generated from data captured by the BW-TOF sensor unit 2A. The first image I1 is a brightness image (black and white image) captured by BW pixels, with each pixel represented by brightness. The second image I2 is a distance image captured by IR pixels, with each pixel represented by distance. The "distance" here refers to the distance from an object captured in the second image I2 to the sensor unit 2 (or camera module 100).

[0022] It should be noted that the sensor having both the functions of a 2D camera and a 3D camera is not limited to the BW-TOF sensor unit 2A. For example, the sensor having both the functions of a 2D camera and a 3D camera may be an RGB-TOF sensor. The RGB-TOF sensor differs from the BW-TOF sensor unit 2A in that it can generate a color image as a luminance image instead of a black-and-white image.

[0023] The first recognition unit 12 executes a first recognition process to recognize the type of a first object Ob1 (see FIG. 6, etc.) appearing in the first image I1. Here, the first recognition process is a process to recognize the first object Ob1 appearing in the first image I1 based on brightness.

[0024] Here, "recognizing the first object Ob1 based on luminance" means identifying whether or not a predetermined object is captured in the first image I1 based on luminance. The object is the target of recognition and is set appropriately depending on the application of the object detection system 1. For example, if it is desired to detect a person using the object detection system 1, the object would be a person. In this case, if a person is captured in the first image I1, the person is recognized as the first object Ob1 in the first recognition process. On the other hand, if a tree is captured in the first image I1, the tree is not recognized as the first object Ob1 in the first recognition process.

[0025] Note that the object to be recognized in the first recognition process is not limited to a person, but may be another object. Furthermore, the object to be recognized in the first recognition process is not limited to one type, but may be multiple types.

[0026] In the embodiment, the first recognition unit 12 performs the first recognition process by using a trained model that has been machine-learned in advance to recognize a predetermined object when the first image I1 is input. Note that the first recognition unit 12 may perform the first recognition process by, for example, performing a pattern matching process on the first image I1 using the predetermined object as a pattern image.

[0027] The second recognition unit 13 executes a second recognition process to recognize the position (particularly, the position in three-dimensional space) of a second object Ob2 (see FIG. 6, etc.) appearing in the second image I2. Here, the second recognition process is a process to recognize the second object Ob2 appearing in the second image I2 based on the distance.

[0028] Here, "recognizing the second object Ob2 based on distance" means grouping multiple pixels that are close in distance (depth value) (i.e., the difference in depth value between pixels is smaller than a threshold) into one group and identifying the group as an object. Therefore, unlike the first recognition process, the second recognition process does not care about the type of object, and the second object Ob2 is not necessarily a predetermined object.

[0029] 2 is an explanatory diagram of a first region A1 in a first image I1 and a second region A2 in a second image I2. In the example shown in Fig. 2, the first image I1 and the second image I2 are assumed to overlap. In addition, in the example shown in Fig. 2, the first object Ob1 and the second object Ob2 are assumed to be the same object.

[0030] The first area A1 is an area based on the first object Ob1 in the first image I1. The first area A1 is set in the first image I1 when the first object Ob1 is recognized in the first recognition process. The second area A2 is an area based on the second object Ob2 in the second image I2. The second area A2 is set in the second image I2 when the second object Ob2 is recognized in the second recognition process.

[0031] In this embodiment, as shown in Fig. 2, the first region A1 is set in the first image I1 as a rectangular region that includes not only a plurality of pixels representing the first object Ob1 but also one or more pixels surrounding the first object Ob1. Also, in this embodiment, as shown in Fig. 2, the second region A2 is set in the second image I2 as a rectangular region that includes not only a plurality of pixels representing the second object Ob2 but also one or more pixels surrounding the second object Ob2.

[0032] 2, the first area A1′ may be set as an area including only a plurality of pixels representing the first object Ob1, and the second area A2′ may be set as an area including only a plurality of pixels representing the second object Ob2.

[0033] The detection unit 14 executes a detection process to detect the first object Ob1 and the second object Ob2 as the same object when a first region A1 based on a first object Ob1 in the first image I1 overlaps with a second region A2 based on a second object Ob2 in the second image I2. The detection process is a so-called fusion process. In the embodiment, the detection process executes a first determination process to determine whether the first object Ob1 and the second object Ob2 are the same object based on whether a first ratio R_2d and a second ratio R_3d exceed reference values. Here, the first ratio R_2d refers to the ratio of an overlap region A3 (see FIG. 3 ) where the first region A1 and the second region A2 overlap in the first region A1. The second ratio R_3d refers to the ratio of the overlap region A3 in the second region A2.

[0034] FIG. 3 is an explanatory diagram of the first ratio R_2d and the second ratio R_3d. As shown in FIG. 3A, the area of ​​the first region A1 is represented by "A_2d," and the area of ​​the second region A2 is represented by "A_3d." As shown in FIG. 3B, the area of ​​the overlap region A3 is represented by "A_int." As shown in FIG. 3C, the area of ​​the entire region including the first region A1 and the second region A2 is represented by "A_uni." In this case, the first ratio R_2d is expressed by the formula "R_2d = A_int / A_2d." The second ratio R_3d is expressed by the formula "R_3d = A_int / A_3d."

[0035] In the detection process, as described above, the first determination process is first performed. In the first determination process, the first object Ob1 and the second object Ob2 are determined to be the same object if they satisfy the conditions that the first ratio R_2d is equal to or greater than the first threshold R_2dth, which is a reference value, and the second ratio R_3d is equal to or greater than the second threshold R_3dth, which is a reference value. On the other hand, in the first determination process, the first object Ob1 and the second object Ob2 are determined not to be the same object if they do not satisfy the above conditions. The first threshold R_2dth and the second threshold R_3dth can be set appropriately depending on the detection accuracy required of the object detection system 1. The first threshold R_2dth is, for example, 0.05. The second threshold R_3dth is, for example, 0.3.

[0036] In the detection process, if there are multiple second objects Ob2 determined to be the same object as the first object Ob1 by the first determination process, a second determination process is executed to determine whether any of the multiple second objects Ob2 is the same object as the first object Ob1, further based on the magnitude of the first ratio R_2d. In the second determination process, for example, the second object Ob2 with the highest first ratio R_2d among the multiple second objects Ob2 is determined to be the same object as the first object Ob1.

[0037] In the detection process, if there are multiple second objects Ob2 determined to be the same object as the first object Ob1 in the second determination process, a third determination process is executed to determine whether any of the multiple second objects Ob2 is the same object as the first object Ob1, further based on the distance. Here, the presence of multiple second objects Ob2 determined to be the same object as the first object Ob1 in the second determination process corresponds to, for example, the presence of multiple second objects Ob2 with almost the same first ratio R_2d. In the third determination process, for example, the second object Ob2 with the shortest distance among the multiple second objects Ob2 is determined to be the same object as the first object Ob1.

[0038] In the third determination process, it may be determined whether any one of the multiple second objects Ob2 and the first object Ob1 are the same object based on a parameter other than distance. For example, in the third determination process, it may be determined that the second object Ob2 whose second region A2 is closest in size to the first region A1 is the same object as the first object Ob1. Furthermore, in the third determination process, it may be determined that the second object Ob2 whose representative value (e.g., average, maximum, or mode) of the luminance values ​​of the multiple pixels included in the second region A2 is the largest is the same object as the first object Ob1. In addition, in the third determination process, it may be determined that any one of the multiple second objects Ob2 and the first object Ob1 are the same object based on at least one of the distance, the sizes of the first region A1 and the second region A2, and the representative value of the luminance values ​​of the multiple pixels included in the second region A2.

[0039] A specific example of the detection process will be described below with reference to FIG. 4 . FIG. 4 is a diagram illustrating a specific example of the detection process performed by the object detection system 1 according to the embodiment. In the example shown in FIG. 4( a), a first image I1 and a second image I2 are overlapped. In addition, in the example shown in FIG. 4( a), two first objects Ob11 and Ob12 are recognized in the first image I1 by the first recognition process. In addition, in the example shown in FIG. 4( a), five second objects Ob21, Ob22, Ob23, Ob24, and Ob25 are recognized in the second image I2 by the second recognition process.

[0040] In the detection process, it is determined whether the first object Ob11 and each of the five second objects Ob21, Ob22, Ob23, Ob24, and Ob25 are the same object. Note that the second regions A23, A24, and A25 based on the three second objects Ob23, Ob24, and Ob25 do not overlap at all with the first region A11 based on the first object Ob11. For this reason, the following description of the determination process for these three second objects Ob23, Ob24, and Ob25 will be omitted, and only the determination process for the two second objects Ob21 and Ob22 will be described, as shown in FIG. 4B.

[0041] First, in the detection process, a first determination process is performed on the combination of the first object Ob11 and the second object Ob21. If the first ratio for this combination is "R_2d_11" and the second ratio is "R_3d_11," then "R_2d_11≧R_2dth," but "R_3d_11<R_3dth," so the condition is not satisfied. Therefore, in the first determination process, it is determined that the first object Ob11 and the second object Ob21 are not the same object.

[0042] Next, in the detection process, a first determination process is performed on the combination of the first object Ob11 and the second object Ob22. If the first ratio for this combination is "R_2d_12" and the second ratio is "R_3d_12," then "R_2d_12≧R_2dth" and "R_3d_12≧R_3dth" are satisfied, and therefore the condition is met. Therefore, in the first determination process, it is determined that the first object Ob11 and the second object Ob22 are the same object. As a result, in the detection process, the first object Ob11 and the second object Ob22 are detected as the same object.

[0043] Furthermore, in the detection process, it is determined whether the first object Ob12 and each of the five second objects Ob21, Ob22, Ob23, Ob24, and Ob25 are the same object. Note that the second regions A21 and A22 based on the two second objects Ob21 and Ob22 do not overlap at all with the first region A12 based on the first object Ob12. For this reason, the following description of the determination process for these two second objects Ob21 and Ob22 will be omitted, and only the determination process for the three second objects Ob23, Ob24, and Ob25 will be described, as shown in FIG. 4C .

[0044] First, in the detection process, a first determination process is performed on the combination of the first object Ob12 and the second object Ob23. If the first ratio for this combination is "R_2d_23" and the second ratio is "R_3d_23," then "R_2d_23<R_2dth" and "R_3d_23<R_3dth," and therefore the condition is not satisfied. Therefore, in the first determination process, it is determined that the first object Ob12 and the second object Ob23 are not the same object.

[0045] Next, in the detection process, a first determination process is performed on the combination of the first object Ob12 and the second object Ob24. If the first ratio for this combination is "R_2d_24" and the second ratio is "R_3d_24," then "R_2d_24≧R_2dth" and "R_3d_24≧R_3dth" are satisfied, and therefore the first determination process determines that the first object Ob12 and the second object Ob24 are the same object.

[0046] Next, in the detection process, a first determination process is performed on the combination of the first object Ob12 and the second object Ob25. If the first ratio for this combination is "R_2d_25" and the second ratio is "R_3d_25," then "R_2d_25≧R_2dth" and "R_3d_25≧R_3dth" are satisfied, and therefore the first determination process determines that the first object Ob12 and the second object Ob25 are the same object.

[0047] Here, because there are two second objects Ob2 that are determined to be the same object as the first object Ob12 by the first determination process, the second determination process is executed in the detection process. Here, when the first ratio "R_2d_24" for the combination of the first object Ob12 and the second object Ob24 is compared with the first ratio "R_2d_25" for the combination of the first object Ob12 and the second object Ob25, "R_2d_24 > R_2d_25" is obtained. Therefore, in the second determination process, it is determined that the first object Ob12 and the second object Ob24 are the same object. As a result, in the detection process, the first object Ob12 and the second object Ob24 are detected as the same object.

[0048] The output unit 15 executes an output process to output object information about an object detected in the detection process by the detection unit 14, including information indicating the type of the object recognized in the first recognition process and information indicating the position of the object recognized in the second recognition process. Specifically, the object information is information obtained by combining the recognition results of a first object Ob1 and a second object Ob2 that have been detected as the same object. The recognition result of the first object Ob1 is information indicating the type of the first object Ob1 recognized in the first recognition process. The recognition result of the second object Ob2 is information indicating the position (position in three-dimensional space) of the second object Ob2 recognized in the second recognition process.

[0049] For example, when the camera module 100 is mounted on a vehicle such as an automobile, the output unit 15 outputs the object information to an in-vehicle ECU (Electronic Control Unit). In this case, the in-vehicle ECU can use the object information to control the vehicle. Also, for example, the output unit 15 outputs the object information to an in-vehicle display for display. In this case, the driver of the vehicle can use the object information to drive the vehicle by looking at the in-vehicle display.

[0050] Furthermore, in the output process by the output unit 15, the object information may further include information indicating the moving speed or moving direction of the object indicated by the object information (i.e., the object detected in the detection process). The moving speed of the object can be calculated by referring to the time-series data of the information indicating the position of the second object Ob2. Similarly, the moving direction of the object can be calculated by referring to the time-series data of the information indicating the position of the second object Ob2.

[0051] The object detection system 1 may further perform a tracking process to track the object indicated by the object information (i.e., the object detected in the detection process). In the tracking process, the object detected in the detection process is tracked to acquire time-series data of the object. Then, in an output process by the output unit 15, the result of the tracking process may be further output.

[0052] For example, if the output unit 15 outputs the results of the tracking process to an in-vehicle ECU, the in-vehicle ECU can control the vehicle based on the trajectory of the detected object. Also, if the output unit 15 outputs the results of the tracking process to an in-vehicle display for display, the driver of the vehicle can drive the vehicle based on the trajectory of the detected object by looking at the in-vehicle display.

[0053] <Operation> An example of the operation of the object detection system 1 according to the embodiment will be described below with reference to FIG.

[0054] First, the acquisition unit 11 of the object detection system 1 acquires a first image I1 and a second image I2 (step S1). In this embodiment, the acquisition unit 11 acquires a luminance image and a range image captured by the BW-TOF sensor unit 2A, which is the sensor unit 2, thereby acquiring the first image I1 and the second image I2.

[0055] Next, the first recognition unit 12 of the object detection system 1 executes a first recognition process to recognize the type of the first object Ob1 in the first image I1 (step S2). The second recognition unit 13 of the object detection system 1 executes a second recognition process to recognize the position of the second object Ob2 in the second image I2 (step S3). Note that steps S2 and S3 may be executed in reverse order. Also, steps S2 and S3 may be executed in parallel.

[0056] Next, the detection unit 14 of the object detection system 1 executes a detection process. Specifically, it determines whether the first object Ob1 recognized in the first recognition process and the second object Ob2 recognized in the second recognition process are the same object (step S4). In this embodiment, the detection unit 14 first executes a first determination process. Then, if there are multiple second objects Ob2 determined to be the same object as the first object Ob1 by the first determination process, the detection unit 14 further executes a second determination process. Furthermore, if there are multiple second objects Ob2 determined to be the same object as the first object Ob1 by the second determination process, the detection unit 14 further executes a third determination process.

[0057] When the first object Ob1 and the second object Ob2 are detected as the same object by the detection process, the object detection system 1 combines the recognition result of the first object Ob1 determined to be the same object with the recognition result of the second object Ob2 (step S5). Step S5 may be executed by the detection unit 14 or the output unit 15.

[0058] Furthermore, the object detection system 1 executes a tracking process as necessary (step S6). Step S6 may be executed by the detection unit 14 or the output unit 15. Whether or not to execute the tracking process can be set as appropriate.

[0059] Then, the output unit 15 of the object detection system 1 executes an output process (step S7). In the embodiment, the output unit 15 outputs object information including at least information indicating the type of object detected in the detection process and information indicating the position of the object.

[0060] 6 is a diagram showing a specific example of object detection by the object detection system 1 according to the embodiment. In the example shown in FIG. 6, the first recognition process by the first recognition unit 12 recognizes a person as a predetermined object.

[0061] 6 , in the first recognition process, a human-shaped first object Ob11 is recognized in the first image I1, but a tree is not recognized as the first object Ob1. Therefore, a first region A11 based on the first object Ob11 is set in the first image I1. Furthermore, in the second recognition process, a human-shaped second object Ob21 and a tree-shaped second object Ob22 are recognized in the second image I2. Therefore, a second region A21 based on the human-shaped second object Ob21 and a second region A22 based on the tree-shaped second object Ob22 are set in the second image I2.

[0062] 6 , the object detection system 1 detects the first human-shaped object Ob11 and the second human-shaped object Ob21 as the same object because the first area A11 and the second area A21 overlap. Therefore, the object detection system 1 outputs object information in which the object type is "person" and the object position is "(x1, y1, z1)." "(x1, y1, z1)" are coordinates indicating the position of the second object Ob21 recognized by the second recognition process.

[0063] 6 , the object detection system 1 also outputs object information about a tree-shaped second object Ob22, even though there is no first area A1 that overlaps with the second area A22. In this object information, the object position is represented as "(x2, y2, z2)" and the object type is represented as "unknown." "(x2, y2, z2)" are coordinates indicating the position of the second object Ob22 recognized by the second recognition process. In this way, the object detection system 1 may output not only information about objects detected as the same object by the detection process, but also information about objects not detected as the same object.

[0064] <Advantages> The advantages of the object detection system 1 according to the embodiment will be described below. The techniques disclosed in Patent Documents 1 and 2 require a projection process in which the camera detection results are projected onto the radar coordinate space or the radar detection results are projected onto an image captured by the camera before performing so-called fusion processing. For this reason, the techniques disclosed in Patent Documents 1 and 2 have problems such as a deterioration in detection accuracy due to errors occurring in the projection process, and an increase in the processing time required to recognize an object due to the time taken to perform the projection process.

[0065] In contrast, in the object detection system 1 according to the embodiment, a first recognition process is performed on the first image I1, and a second recognition process is performed on the second image I2, in which each pixel corresponds one-to-one to each pixel of the first image I1, and then the detection process (fusion process) is performed. Therefore, in the object detection system 1 according to the embodiment, the first area A1 and the second area A2 can be set in the first image I1 and the second image I2, respectively, and the projection process described above is not necessary. Therefore, the object detection system 1 according to the embodiment does not encounter any issues associated with performing the projection process, which has the advantage of easily reducing processing time while improving detection accuracy.

[0066] While the object detection system according to one or more aspects of the present disclosure has been described above based on the embodiments, the present disclosure is not limited to these embodiments. As long as they do not deviate from the spirit of the present disclosure, various modifications conceivable by a person skilled in the art to each embodiment and configurations constructed by combining components of different embodiments may also be included within the scope of one or more aspects of the present disclosure.

[0067] 7 is a diagram showing a specific example of the detection process performed by the object detection system 1 according to a first modification of the embodiment. The detection process performed by the object detection system 1 according to the first modification is different from the detection process performed by the object detection system 1 according to the embodiment. Note that, below, a description of the points common to the object detection system 1 according to the embodiment will be omitted.

[0068] As shown in the example of Figure 7A, when an object with a complex shape, such as agricultural machinery, is captured in the second image I2, the second recognition process may recognize the object as multiple second objects Ob2 instead of a single second object Ob2. In the example of Figure 7A, the first recognition process recognizes the agricultural machinery-type object as a single first object Ob11. However, the second recognition process does not recognize the agricultural machinery-type object as a single second object Ob2, but instead recognizes it as four second objects Ob21, Ob22, Ob23, and Ob24.

[0069] In this case, the second regions A21, A22, A23, and A24 based on the second objects Ob21, Ob22, Ob23, and Ob24, respectively, are smaller than the second region A2 when the agricultural machinery-type object is the single second object Ob2. This reduces the first ratio R_2d of each of the second objects Ob21, Ob22, Ob23, and Ob24, which makes it difficult to determine that the second objects are the same object as the first object Ob11 in the first determination process.

[0070] Therefore, in the first modified example, if a plurality of second objects Ob2 are recognized in the second recognition process, if the second region A2 based on each of the plurality of second objects Ob2 overlaps with the first region A1 based on the same first object Ob1, and if the distance between each of the plurality of second objects Ob2 falls within a predetermined range, the plurality of second objects Ob2 are combined. Then, in the detection process, it is determined whether the combined second object Ob2 and the first object Ob1 are the same object.

[0071] First, in the detection process, for each combination of the first object Ob11 and each of the plurality of second objects Ob2, two or more second objects Ob2 are determined, each of which has a second ratio R_3d equal to or greater than the second threshold R_3dth. In the example shown in FIG. 7A, four second objects Ob21, Ob22, Ob23, and Ob24 correspond to the two or more second objects Ob2. In the example shown in FIG. 7A, second regions A21, A22, A23, and A24 based on the four second objects Ob21, Ob22, Ob23, and Ob24 overlap with the first region A11 based on the same first object Ob11. Furthermore, the distances of the plurality of second objects Ob21, Ob22, Ob23, and Ob24 all fall within a predetermined range.

[0072] Therefore, in the detection process, as shown in (b) of FIG. 7, for example, these four second objects Ob21, Ob22, Ob23, and Ob24 are combined, and a second region A21' is set in the second image I2 based on a single second object Ob21' that includes these four second objects Ob21, Ob22, Ob23, and Ob24. Note that in the example shown in (b) of FIG. 7, the second region A21' is a rectangular region that includes the second regions A21, A22, A23, and A24, but is not limited to this. For example, the second region A21' may be a region that includes only the second regions A21, A22, A23, and A24.

[0073] Thereafter, the detection unit 14 performs a first determination process on the combination of the first object Ob11 and the combined second object Ob21'. This enables the detection unit 14 to detect the first object Ob11 and the combined second object Ob21' as the same object. Note that if there are multiple combined second objects Ob2 that have been determined to be the same object as the first object Ob1 by the first determination process, the detection unit 14 further performs a second determination process. Furthermore, if there are multiple combined second objects Ob2 that have been determined to be the same object as the first object Ob1 by the second determination process, the detection unit 14 further performs a third determination process.

[0074] Here, in the detection process, the position of any one of the multiple second objects Ob2 that were the subject of synthesis may be calculated as the position of the detected object. For example, the position of the second object Ob2 that is closest among the multiple second objects Ob2 may be calculated as the position of the detected object. Furthermore, for example, the position of the second object Ob2 that has the largest second region A2 among the multiple second objects Ob2 may be calculated as the position of the detected object. Furthermore, for example, the position of the second object Ob2 that has the largest number of pixels recognized as the second object Ob2 among the multiple second objects Ob2 may be calculated as the position of the detected object. Furthermore, for example, the position of the second object Ob2 that has the highest brightness of pixels recognized as the second object Ob2 among the multiple second objects Ob2 may be calculated as the position of the detected object.

[0075] In this case, the position of the detected object may be calculated as one of the coordinates of the closest point from the sensor unit 2 of the second object Ob2 selected as described above, the far coordinates of the farthest point from the sensor unit 2, and the center of gravity coordinates.

[0076] In addition, in the detection process, the position of the synthesized second object Ob2 may be calculated as the position of the detected object. In this case, the position of the detected object may be calculated as any one of the coordinates of the synthesized second object Ob2's closest point from the sensor unit 2, the far coordinates of the farthest point from the sensor unit 2, and the barycentric coordinates.

[0077] 8 is a block diagram showing an overview of an object detection system 1A according to a second modification of the embodiment. The object detection system 1A according to the second modification differs from the object detection system 1 according to the embodiment in that the sensor unit 2 is a TOF sensor unit 2B rather than a BW-TOF sensor unit 2A. Note that, below, a description of the points common to the object detection system 1 according to the embodiment will be omitted.

[0078] The TOF sensor unit 2B is a distance sensor. The TOF sensor unit 2B includes a TOF sensor 21B and a light source 22. The TOF sensor 21B has pixels (IR pixels) that receive near-infrared light and can generate a near-infrared light luminance image (IR luminance image) and a distance image. Therefore, the acquisition unit 11 of the object detection system 1A according to the second modification acquires, as the first image I1, a luminance image (IR luminance image) in which each pixel is represented by luminance, and acquires, as the second image I2, a distance image in which each pixel is represented by distance.

[0079] <Other Modifications> For example, in the embodiments, the first determination process may determine whether the first object Ob1 and the second object Ob2 are the same object based on a third ratio IoU, which is the ratio of the overlapping region A3 to the entire region including the first region A1 and the second region A2. The third ratio IoU is expressed by the formula "IoU = A_int / A_uni." For example, the first determination process may determine that the first object Ob1 and the second object Ob2 are the same object if the third ratio IoU is equal to or greater than a third threshold IoU_th, which is a reference value, and may determine that the first object Ob1 and the second object Ob2 are not the same object if the third ratio IoU does not satisfy the above condition.

[0080] Furthermore, in the first determination process, it may be determined whether the first object Ob1 and the second object Ob2 are the same object by combining the first ratio R_2d, the second ratio R_3d, and the third ratio IoU. That is, in the detection process, it may be determined whether the first object Ob1 and the second object Ob2 are the same object based on whether at least one of the first ratio R_2d, the second ratio R_3d, and the third ratio IoU exceeds a reference value.

[0081] For example, in an embodiment, if there are multiple second objects Ob2 that are determined to be the same object as the first object Ob1 in the first judgment process, the second judgment process may determine whether any of the multiple second objects Ob2 is the same object as the first object Ob1, further based on the magnitude of at least one of the first ratio R_2d, the second ratio R_3d, and the third ratio IoU.

[0082] For example, in the embodiment, the first image I1 and the second image I2 may be a single image. For example, if the sensor unit 2 is a TOF sensor, the distance image output by the TOF sensor may be the first image I1 and the second image I2. In other words, the distance image is both the first image I1 and the second image I2. In this case, the first recognition process and the second recognition process may be performed on the distance image.

[0083] For example, in the embodiment, the object detection system 1 is mounted on the camera module 100, but this is not limiting. For example, the object detection system 1 may be realized by a device other than the camera module 100, such as a personal computer.

[0084] For example, the present disclosure can be realized not only as an object detection system, but also as an object detection method including steps (processing) performed by components that make up the object detection system.

[0085] 5 , the object detection method includes acquiring a first image I1 and a second image I2, each pixel of which corresponds one-to-one to a pixel in the first image I1 (step S1), performing a first recognition process to recognize the type of a first object Ob1 appearing in the first image I1 (step S2), performing a second recognition process to recognize the position of a second object Ob2 appearing in the second image I2 (step S3), and performing a detection process to detect the first object Ob1 and the second object Ob2 as the same object when a first area A1 defined based on the first object Ob1 in the first image I1 and a second area A2 defined based on the second object Ob2 in the second image I2 overlap (step S4).

[0086] Furthermore, as described above, when the first image I1 and the second image I2 are one image, the object detection method acquires the one image, performs a first recognition process to recognize the type of first object Ob1 appearing in the one image, performs a second recognition process to recognize the position of second object Ob2 appearing in the one image, and when a first area A1 based on the first object Ob1 in the one image and a second area A2 based on the second object Ob2 in the one image overlap, performs a detection process to detect the first object Ob1 and the second object Ob2 as the same object.

[0087] For example, the present disclosure can be realized as a program for causing a computer (processor) to execute steps included in the object detection method. The processor may be one or more. Furthermore, the present disclosure can be realized as a non-transitory computer-readable recording medium, such as a CD-ROM, on which the program is recorded.

[0088] For example, when the present disclosure is realized as a program (software), each step is performed by running the program using hardware resources such as a computer's CPU, memory, input / output circuitry, etc. In other words, each step is performed by the CPU acquiring data from memory or input / output circuitry, etc., performing calculations on the data, and outputting the calculation results to memory or input / output circuitry, etc.

[0089] In the above-described embodiment, each component included in the object detection system may be configured with dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0090] Some or all of the functions of the object detection system according to the above-described embodiments are typically realized as an LSI, which is an integrated circuit. These may be individually integrated into single chips, or some or all of them may be integrated into a single chip. Furthermore, the integrated circuit is not limited to an LSI, and may be realized using a dedicated circuit or a general-purpose processor. It is also possible to use an FPGA (Field Programmable Gate Array), which can be programmed after LSI manufacturing, or a reconfigurable processor, which can reconfigure the connections and settings of circuit cells within an LSI.

[0091] (Additional Notes) The above description of the embodiments discloses the following techniques.

[0092] (Technology 1) An object detection method comprising: acquiring a first image and a second image in which each pixel corresponds one-to-one to each pixel of the first image; performing a first recognition process to recognize the type of a first object appearing in the first image; performing a second recognition process to recognize the position of a second object appearing in the second image; and performing a detection process to detect the first object and the second object as the same object when a first area in the first image based on the first object and a second area in the second image based on the second object overlap.

[0093] This allows the first and second areas to be set in the first and second images, respectively, eliminating the need for projection processing to project the camera's detection results onto the radar's coordinate space or to project the radar's detection results onto the image captured by the camera. Therefore, this has the advantage of eliminating the issues that arise from executing projection processing, making it easier to reduce processing time while improving detection accuracy.

[0094] (Technology 2) The object detection method according to Technology 1, wherein the first image and the second image are both images generated from data captured by a single sensor unit.

[0095] This has the advantage that, compared to when the first image and the second image are obtained from data captured by multiple sensor units, it is easier to achieve a one-to-one correspondence between each pixel of the first image and each pixel of the second image.

[0096] (Technology 3) An object detection method described in Technology 2, wherein the sensor unit is a sensor composed of pixels that receive near-infrared light, the first image is a luminance image in which each pixel is represented by the luminance of the near-infrared light, and the second image is a distance image in which each pixel is represented by a distance calculated from the amount of received near-infrared light.

[0097] This has the advantage that the first and second images can be acquired without using a 2D camera.

[0098] (Technology 4) The object detection method described in Technology 2, wherein the sensor unit is a sensor in which pixels that receive visible light and pixels that receive near-infrared light are integrally arranged, the first image is a luminance image captured by the pixels that receive the visible light, with each pixel represented by the luminance of the visible light, and the second image is a distance image captured by the pixels that receive the near-infrared light, with each pixel represented by a distance calculated from the amount of near-infrared light received.

[0099] This has the advantage that the 2D camera and the 3D camera are integrally provided, making it easier to reduce the size of the sensor unit.

[0100] (Technology 5) An object detection method that acquires an image, performs a first recognition process that recognizes the type of a first object that appears in the image, performs a second recognition process that recognizes the position of a second object that appears in the image, and performs a detection process that detects the first object and the second object as the same object when a first area in the image based on the first object and a second area in the image based on the second object overlap.

[0101] This allows the first and second regions to be set in a single image, eliminating the need for projection processing to project the camera's detection results onto the radar's coordinate space or to project the radar's detection results onto the image captured by the camera. Therefore, this has the advantage of eliminating the issues that arise from executing projection processing, making it easier to reduce processing time while improving detection accuracy.

[0102] (Technology 6) An object detection method described in any of Technologies 1 to 5, in which the detection process performs a first determination process to determine whether the first object and the second object are the same object based on whether at least one of the following proportions exceeds a reference value: a first proportion of the overlapping area where the first area and the second area overlap in the first area; a second proportion of the overlapping area in the second area; and a third proportion, which is the proportion of the overlapping area in the entire area including the first area and the second area.

[0103] This has the advantage that a first object and a second object that meet the condition that at least one of the first ratio, second ratio, and third ratio exceeds a reference value are detected as the same object, making it easier to improve detection accuracy.

[0104] (Technology 7) In the object detection method described in Technology 6, if there are multiple second objects determined to be the same object as the first object in the first determination process, a second determination process is executed to determine whether any of the multiple second objects is the same object as the first object based on the magnitude of at least one of the first ratio, the second ratio, and the third ratio.

[0105] This has the advantage that when there are multiple second objects that are determined to be the same object as the first object, one of the second objects can be selected based on the magnitude of at least one of the first ratio, second ratio, and third ratio, making it easier to improve detection accuracy.

[0106] (Technology 8) In the object detection method described in Technology 7, if there are multiple second objects determined to be the same object as the first object in the second determination process, a third determination process is executed to determine whether any of the multiple second objects is the same object as the first object based on at least one of the distance, the size of the first region and the second region, and a representative value of the brightness values ​​of multiple pixels included in the second region.

[0107] This has the advantage that when there are multiple second objects that are determined to be the same object as the first object, one of the second objects can be selected based on at least one of the distance, the size of the first area and the second area, and the representative value of the brightness values ​​of the multiple pixels contained in the second area, making it easier to improve detection accuracy.

[0108] (Technology 9) In the object detection method described in any of Technologies 1 to 8, if the second objects recognized in the second recognition process are multiple, and the second areas based on each of the multiple second objects overlap with the first area based on the same first object, and the distance between each of the multiple second objects falls within a predetermined range, the multiple second objects are combined, and it is determined whether the combined second object and the first object are the same object.

[0109] This has the advantage that even if an object that is actually a single object is recognized as multiple second objects, the multiple second objects can be detected as a single object, making it easier to improve detection accuracy.

[0110] (Technology 10) An object detection method described in Technology 9, in which the detection process calculates the position of any one of the multiple second objects that are the subject of synthesis as the position of the detected object.

[0111] This has the advantage that the accuracy of calculating the position of the detected object can be easily improved because the position of any one of the multiple second objects that are the subject of synthesis is referenced.

[0112] (Technology 11) The object detection method according to Technology 9, wherein the detection process calculates the position of the synthesized second object as the position of the detected object.

[0113] This has the advantage that the accuracy of calculating the position of the detected object can be easily improved by referring to the position of the synthesized second object.

[0114] (Technology 12) An object detection method described in any one of Technologies 1 to 11, further performing an output process to output object information regarding the object detected in the detection process, including information indicating the type of the object recognized in the first recognition process and information indicating the position of the object recognized in the second recognition process.

[0115] This has the advantage that the type and position of the object detected by the detection process can be referenced in the output destination system, etc., making it easier to execute control based on the type and position of the object.

[0116] (Technology 13) The object detection method according to Technology 12, wherein the output process further includes outputting information indicating the moving speed or moving direction of the object indicated by the object information, the information being further included in the object information.

[0117] This has the advantage that the movement speed or movement direction of the object detected by the detection process can be referenced in the output system, etc., making it easier to carry out control based on the movement speed or movement direction of the object.

[0118] (Technology 14) The object detection method according to Technology 12 or 13, further comprising: performing a tracking process for tracking the object indicated by the object information; and outputting a result of the tracking process in the output process.

[0119] This has the advantage that the trajectory of the object detected by the detection process can be referenced in the output destination system, making it easier to execute control based on the trajectory of the object.

[0120] (Technology 15) A program that causes one or more processors to execute the object detection method according to any one of technologies 1 to 14.

[0121] This makes it possible to provide a program that can easily reduce processing time while improving detection accuracy.

[0122] (Technology 16) An object detection system comprising: an acquisition unit that acquires a first image and a second image in which each pixel corresponds one-to-one to each pixel of the first image; a first recognition unit that performs a first recognition process that recognizes the type of a first object appearing in the first image; a second recognition unit that performs a second recognition process that recognizes the position of a second object appearing in the second image; and a detection unit that performs a detection process that detects the first object and the second object as the same object when a first area in the first image based on the first object and a second area in the second image based on the second object overlap.

[0123] This makes it possible to provide an object detection system that can easily reduce processing time while improving detection accuracy.

[0124] The present disclosure can be applied to devices that recognize objects using so-called fusion processing.

[0125] 1, 1A Object detection system 11 Acquisition unit 12 First recognition unit 13 Second recognition unit 14 Detection unit 15 Output unit 2 Sensor unit 2A BW-TOF sensor unit 21 BW-TOF sensor 22 Light source 2B TOF sensor unit 21B TOF sensor 100 Camera module A1, A1', A11, A12 First region A2, A2', A21, A22, A23, A24, A25 Second region A21' Synthesized second region A3 Overlap region A_2D Area of ​​first region A_3D Area of ​​second region A_int Area of ​​overlap region A_uni Area of ​​total region I1 First image I2 Second image Ob1, Ob11, Ob12 First object Ob2, Ob21, Ob22, Ob23, Ob24, Ob25 Second object Ob21' Composite second object

Claims

1. acquiring a first image and a second image, each pixel of which corresponds one-to-one to each pixel of the first image; executing a first recognition process for recognizing a type of a first object captured in the first image; executing a second recognition process for recognizing a position of a second object appearing in the second image; executing a detection process for detecting the first object and the second object as a same object when a first region based on the first object in the first image and a second region based on the second object in the second image overlap; Object detection methods.

2. The first image and the second image are both images generated from data captured by one sensor unit. The object detection method according to claim 1 .

3. The sensor unit is a sensor configured with pixels that receive near-infrared light, the first image is a luminance image in which each pixel is represented by the luminance of the near-infrared light, The second image is a distance image in which each pixel is represented by a distance calculated from the amount of received near-infrared light. The object detection method according to claim 2 .

4. The sensor unit is a sensor in which a pixel for receiving visible light and a pixel for receiving near-infrared light are integrally provided, the first image is a luminance image captured by pixels that receive the visible light, and each pixel is represented by a luminance of the visible light; The second image is a distance image captured by pixels that receive the near-infrared light, and each pixel is represented by a distance calculated from the amount of the received near-infrared light. The object detection method according to claim 2 .

5. Take one image, executing a first recognition process for recognizing a type of a first object captured in the one image; executing a second recognition process for recognizing a position of a second object captured in the one image; executing a detection process for detecting the first object and the second object as a same object when a first region in the one image based on the first object and a second region in the one image based on the second object overlap; Object detection methods.

6. In the detection process, a first determination process is executed to determine whether the first object and the second object are the same object based on whether at least one of a first ratio of an overlapping area, where the first area and the second area overlap, to the first area, a second ratio of the overlapping area to the second area, and a third ratio, which is a ratio of the overlapping area to the entire area including the first area and the second area, exceeds a reference value. The object detection method according to any one of claims 1 to 5.

7. In the detection process, when the number of the second objects determined to be the same object as the first object in the first determination process is multiple, a second determination process is executed to determine whether or not any of the second objects among the multiple second objects is the same object as the first object, based on the magnitude of at least one of the first ratio, the second ratio, and the third ratio. The object detection method according to claim 6.

8. In the detection process, when the number of the second objects determined to be the same object as the first object in the second determination process is multiple, a third determination process is executed to determine whether or not any of the second objects among the multiple second objects is the same object as the first object based on at least one of the distance between the multiple second objects, the size of the first region and the second region, and a representative value of the luminance values ​​of the multiple pixels included in the second region. The object detection method according to claim 7.

9. In the detection process, if the number of the second objects recognized in the second recognition process is multiple, and a second region based on each of the multiple second objects overlaps with a first region based on the same first object, and a distance between each of the multiple second objects falls within a predetermined range, the multiple second objects are synthesized, and it is determined whether the synthesized second object and the first object are the same object. The object detection method according to any one of claims 1 to 5.

10. In the detection process, a position of any one of the plurality of second objects that are targets of synthesis is calculated as a position of a detected object. The object detection method according to claim 9.

11. In the detection process, a position of the synthesized second object is calculated as a position of the detected object. The object detection method according to claim 9.

12. further performing an output process of outputting object information related to the object detected in the detection process, including information indicating a type of the object detected in the detection process and information indicating a position of the object detected in the detection process; The object detection method according to any one of claims 1 to 5.

13. In the output process, information indicating a moving speed or a moving direction of the object indicated by the object information is further included in the object information and is output. The object detection method according to claim 12.

14. further performing a tracking process for tracking the object indicated by the object information; The output process further includes outputting a result of the tracking process. The object detection method according to claim 12.

15. One or more processors, Executing the object detection method according to any one of claims 1 to 5, program.

16. an acquisition unit that acquires a first image and a second image in which each pixel corresponds one-to-one to each pixel of the first image; a first recognition unit that executes a first recognition process to recognize a type of a first object captured in the first image; a second recognition unit that executes a second recognition process to recognize a position of a second object captured in the second image; a detection unit that executes a detection process to detect the first object and the second object as a same object when a first region based on the first object in the first image and a second region based on the second object in the second image overlap. Object detection system.