Object recognition device and method, and computer program

Abstract

To improve the responsibility of object recognition.SOLUTION: An object recognition device (100) comprises: recognition means (120) which recognizes an object by performing recognition processing in a second cycle longer than a first cycle to a portion of a plurality of temporary-consecutive images sequentially acquired in the first cycle; and tracking means (130) which performs tracking processing that tracks the object by using the first image applied with the recognition processing and the image following the first image among the plurality of images.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to the technical field of an object recognition device and method for detecting an object from an image, and a computer program. [Background technology]

[0002] As this type of device, for example, a device that recognizes objects in moving images using template matching has been proposed (see Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2009-129388 A Summary of the Invention [Problem to be solved by the invention]

[0004] The technology described in Patent Document 1 has a technical problem in that the time required for image processing related to object recognition becomes a bottleneck, making it difficult to improve responsiveness.

[0005] The present invention has been made in consideration of the above problems, and an object of the present invention is to provide an object recognition device, method, and computer program that can improve the responsiveness of object recognition. [Means for solving the problem]

[0006] An object recognition device according to one aspect of the present invention includes: An object recognition device, A recognition means for recognizing an object by performing a recognition process on a part of a plurality of temporally consecutive images sequentially acquired in a first period, in a second period longer than the first period. Equipped with , The recognition means performs the recognition process on a first image among the plurality of images, and the recognition means performs the recognition process on a second image acquired after a first recognition result is obtained by the recognition process on the first image, and the object recognition device Among the plurality of images, The first recognition result; before Record number A tracking means for performing a tracking process for tracking the object using one image and an image subsequent to the first image. before a second recognition result is obtained by the recognition process on the second image, The tracking means includes: The position of the object is determined at the time when the second image is acquired by tracking the object using the first recognition result, the first image, the second image, and one or more images between the first image and the second image, and before a second recognition result is obtained by the recognition process for the second image, the tracking means determines the position of the object at the time when the second image is acquired by tracking the object using the determined position of the object at the time when the second image is acquired, the second image, and a third image following the second image. That is it.

[0007] One aspect of the object recognition method according to the present invention is an object recognition method for an object recognition device including a recognition means and a tracking means, the object recognition means performing a recognition process on a part of a plurality of temporally consecutive images acquired successively in a first period, the part being a second period longer than the first period, to recognize an object, and in the recognition process, the recognition means performs the recognition process on a first image among the plurality of images, and in the recognition step, the recognition means performs the recognition process on a second image acquired after a first recognition result is obtained by the recognition process on the first image; The object recognition method includes: the tracking means detecting the first recognition result and a previous recognition result from among the plurality of images. Record number a tracking step of performing a tracking process for tracking the object using the first image and an image subsequent to the first image. Further including , In the tracking step, before a second recognition result is obtained by the recognition processing on the second image, the tracking means determines a position of the object at a time when the second image is acquired by tracking the object using the first recognition result, the first image, the second image, and one or more images between the first image and the second image; and in the tracking step, before a second recognition result is obtained by the recognition processing on the second image, the tracking means determines a position of the object at a time when the third image is acquired by tracking the object using the determined position of the object at the time when the second image is acquired, the second image, and a third image following the second image. That is it.

[0008] One aspect of the computer program of the present invention is a computer program for an object recognition device, comprising: A computer program that functions as a recognition means and a tracking means, the recognition means comprising: A part of a plurality of temporally consecutive images sequentially acquired in a first period is subjected to a recognition process in a second period longer than the first period to recognize an object. death , The recognition means performs the recognition process on a first image among the plurality of images, and the recognition means performs the recognition process on a second image acquired after a first recognition result is obtained by the recognition process on the first image, and the tracking means Among the plurality of images, The first recognition result; before Record number A tracking process is performed to track the object using the first image and an image subsequent to the first image. stomach , Before a second recognition result is obtained by the recognition processing for the second image, the tracking means determines a position of the object at a time when the second image is acquired by tracking the object using the first recognition result, the first image, the second image, and one or more images between the first image and the second image, and before a second recognition result is obtained by the recognition processing for the second image, the tracking means determines a position of the object at a time when the third image is acquired by tracking the object using the determined position of the object at the time when the second image is acquired, the second image, and a third image following the second image. That is it. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing a configuration of an object recognition device according to an embodiment. [Diagram 2] 4 is a flowchart illustrating an object recognition process according to the embodiment. [Diagram 3] 10 is a flowchart showing a tracking process according to the embodiment. [Figure 4] FIG. 1 is a diagram illustrating a concept of object recognition according to an embodiment. [Diagram 5] FIG. 1 is a block diagram showing a configuration of a computer according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] An embodiment of an object recognition device will be described with reference to Figs. 1 to 4. An object recognition device according to the embodiment is an object recognition device 100 mounted on a vehicle 1. In Fig. 1, the vehicle 1 includes a camera 10 and the object recognition device 100. The camera 10 is configured to be able to capture an image of, for example, an area in front of the vehicle 1. Note that the vehicle 1 may include one or more cameras in addition to the camera 10.

[0011] The object recognition device 100 comprises an image assigner 110, a recognizer 120, a tracker 130 and an image holder 140, either as logic blocks logically implemented therein or as processing circuits physically implemented therein.

[0012] An image captured by the camera 10 is input to the image allocation unit 110 and the tracking unit 130, and is also temporarily stored in the image storage unit 140. The image stored in the image storage unit 140 is acquired by the tracking unit 130 at a predetermined timing.

[0013] The image assigner 110 determines which images, among those captured by the camera 10, are to be assigned to object recognition processing in the recognizer 120. The image assigner 110 outputs the determined images to the recognizer 120.

[0014] The recognizer 120 performs object recognition processing on the images output from the image allocator 110 to recognize the target object. The target object may be, for example, another vehicle, a pedestrian, a bicycle, or an object fallen on the road. Note that since various existing aspects can be applied to the object recognition processing, detailed explanations thereof will be omitted. Also, what is considered to be the target object may be appropriately set according to, for example, specifications.

[0015] The recognizer 120 outputs a recognition result, which is a result of the object recognition process, to the tracker 130. The recognition result includes information related to the recognized object. The information related to the object may be, for example, information indicating the coordinates of the object in the image, information indicating the range that the object occupies in the image (for example, a circumscribing rectangle of the object, etc.), information indicating the type of the object, etc.

[0016] The tracker 130 is configured to be able to execute a tracking process. When there is the above-mentioned recognition result, the tracker 130 obtains a displacement amount of the object by comparing a first image related to the recognition result (i.e., an image corresponding to an image subjected to object recognition processing by the recognizer 120) with a second image temporally subsequent to the first image. The tracker 130 obtains a position of the object in the second image based on the position of the object in the first image indicated by the recognition result and the above-mentioned displacement amount.

[0017] For example, if there is no recognition result as described above for the image at time Ti, but the position of the object in the image at time Ti is obtained using the image at time Ti and an image at time T(i-1) temporally earlier than the image at time Ti, the tracker 130 obtains the amount of displacement of the object by comparing the image at time Ti with the image at time T(i+1) temporally later than the image at time Ti. The tracker 130 obtains the position of the object in the image at time T(i+1) based on the position of the object in the image at time Ti and the amount of displacement.

[0018] The tracker 130 outputs the determined position of the target object as an object recognition result. Note that, since the method of determining the amount of displacement and the like (i.e., tracking process) described above can be implemented in various existing manners, detailed description thereof will be omitted.

[0019] Next, the operation of the object recognition device 100 will be described with reference to the flowcharts of FIGS.

[0020] 2, the recognizer 120 of the object recognition device 100 acquires an image output from the image assigner 110 (step S101). Next, the recognizer 120 performs object recognition processing on the acquired image to recognize an object (step S102). Note that if an object is not recognized in the processing of step S102 (i.e., if an object does not exist around the vehicle 1), the operation shown in FIG. 2 may be terminated. Thereafter, the recognizer 120 may perform the processing of step S101.

[0021] After the process of step S102, the recognizer 120 outputs the recognition result to the tracker 130 (step S103). At this time, the recognizer 120 may request the image allocator 110 to provide an image to be subjected to the object recognition process next. After that, the recognizer 120 performs the process of step S101. That is, the operation shown in FIG. 2 is periodically repeated.

[0022] 3, the tracker 130 of the object recognition device 100 determines whether or not there is a recognition result (step S201). If it is determined in the process of step S201 that there is a recognition result (step S201: Yes), the tracker 130 acquires a first image related to the recognition result and a second image temporally subsequent to the first image (step S202). At this time, the tracker 130 may acquire the first and second images from the image holder 140.

[0023] Next, the tracker 130 performs a tracking process using the first and second images and the recognition result (step S203). The tracker 130 outputs the result of the process of step S203 as an object recognition result (step S204). After that, the tracker 130 performs the process of step S201. That is, the operation shown in FIG. 3 is periodically repeated.

[0024] If it is determined in the process of step S201 that there is no recognition result (step S201: No), the tracker 130 determines whether or not there is a result of the previous tracking process (i.e., an object recognition result) (step S205). If it is determined in the process of step S205 that there is no result of the previous tracking process (step S205: No), the operation shown in Fig. 3 is terminated. Thereafter, the tracker 130 performs the process of step S201.

[0025] In the process of step S205, if it is determined that there is a result of the previous tracking process (step S205: Yes), the tracker 130 acquires a third image (e.g., the image at the time Ti described above) on which the previous tracking process was performed, and a fourth image (e.g., the image at the time T(i+1) described above) that is temporally later than the third image (step S206). At this time, the tracker 130 may acquire the third image from the image holder 140 and the fourth image from the camera 10, for example.

[0026] Next, the tracker 130 performs a tracking process using the third and fourth images and the result of the previous tracking process (step S207). The tracker 130 outputs the result of the process of step S207 as an object recognition result (step S204). After that, the tracker 130 performs the process of step S201.

[0027] (Technical effect) The technical effect of the object recognition device 100 will be described with reference to FIG. 4. The time required for the above-mentioned object recognition process is, for example, 100 milliseconds. In other words, after an image is captured by the camera 10, the result of the object recognition process is obtained, for example, 100 milliseconds later. On the other hand, when at least one of the camera 10 and the object is moving, the relative positional relationship between the camera 10 and the object changes over time. In this case, a deviation occurs between the position of the object recognized by the object recognition process and the actual position of the object. The greater the relative speed between the camera 10 and the object, the greater the degree of this deviation.

[0028] In order to reduce the degree of the deviation, it is necessary to improve the responsiveness of object recognition. However, due to the nature of the processing, it is extremely difficult to significantly shorten the time required for object recognition processing. Even if the time required for object recognition processing can be shortened by using a processor with a relatively high processing capacity, a problem occurs in that the product cost increases significantly. In other words, the time required for object recognition processing is a bottleneck in the responsiveness of object recognition.

[0029] The time required for tracking processing is significantly shorter than that required for object recognition processing (for example, 1 millisecond). However, tracking processing has the following problem. For example, the position of an object at time t is X, the displacement in an infinitesimal period Δt1 obtained by tracking processing is ΔX1, and the displacement in an infinitesimal period Δt2 following the infinitesimal period Δt1 is ΔX2. In this case, the position of the object at the end of the infinitesimal period Δt2 is "X+ΔX1+ΔX2". Here, there is an error in each of the displacements ΔX1 and ΔX2. For this reason, the error accumulates every time the displacement obtained by tracking processing is added to the position of the object. Therefore, if tracking processing is performed repeatedly, the error in the position of the object becomes large.

[0030] Therefore, in the object recognition device 100, object recognition processing and tracking processing are used in combination. A specific description will be given with reference to FIG. 4. The camera 10 periodically captures images at time intervals shorter than the time required for object recognition processing, and outputs a plurality of images that are continuous in time. For example, it is assumed that the recognizer 120 performs object recognition processing on an image (1) captured at time t1, and the recognition result is obtained between time t4 and time t5. At this time, images (1), (2), (3), and (4) captured at times t1, t2, t3, and t4, respectively, are held in the image holder 140.

[0031] After acquiring the recognition result for image (1), the tracker 130 obtains the displacement of the object between time t1 and time t2 from images (1) and (2), obtains the displacement of the object between time t2 and time t3 from images (2) and (3), and obtains the displacement of the object between time t3 and time t4 from images (3) and (4). The tracker 130 also obtains the displacement of the object between time t4 and time t5 from image (4) and image (5) captured at time t5. The tracker 130 then obtains the position of the object at time t5 from the recognition result for image (1) and the obtained displacement.

[0032] After completing object recognition processing on image (1), recognizer 120 performs object recognition processing on image (5). Until a recognition result is obtained by object recognition processing on image (5), tracker 130 obtains the displacement of the object between time t5 and time t6, for example, from image (5) and image (6) captured at time t6. Then, tracker 130 obtains the position of the object at time t5 and the obtained displacement, and obtains the position of the object at time t6.

[0033] Thereafter, for example, when a recognition result for image (5) is obtained between time t8 and time t9, the tracker 130 obtains the position of the object at time t9 from the recognition result for image (5), image (5), image (6), image (7) captured at time t7, image (8) captured at time t8, and image (9) captured at time t9. Note that, from Fig. 4, if the period during which the camera 10 captures a plurality of temporally consecutive images is defined as a first period, then the period during which the recognizer 120 performs object recognition processing can be said to be a second period that is longer than the first period.

[0034] In this way, by using the object recognition process and the tracking process in combination, for example, the position of an object in an image (5) captured at time t5 can be obtained a very short time after time t5. Therefore, the object recognition device 100 can improve the responsiveness of the object recognition. In addition, since the recognition result of the object recognition process in the recognizer 120 is obtained periodically, it is possible to suppress an increase in errors related to the position of the object. Therefore, the object recognition device 100 can appropriately recognize the object while improving the responsiveness of the object recognition.

[0035] <Computer Program> An embodiment of a computer program will be described with reference to Fig. 5. Fig. 5 is a block diagram showing the configuration of a computer according to the embodiment.

[0036] 5, a computer 50 includes a CPU (Central Processing Unit) 51, a RAM 52, a HDD (Hard Disk Drive) 53, and an I / O 54. The CPU 51, the RAM 52, the HDD 53, and the I / O 54 are connected to one another via a bus 55. A computer program 531 according to this embodiment is stored in advance in the HDD 53. At least one of the RAM 52 and the HDD 53 may constitute the image holder 140 described above.

[0037] The processing of the CPU 51 by the computer program 531 will be described. The CPU 51 acquires an image captured by the camera 10 via the I / O 54. The CPU 51 stores the acquired image in at least one of the RAM 52 and the HDD 53. The CPU 51 determines which of the acquired images should be subjected to object recognition processing. That is, the CPU 51 assigns the images to the object recognition processing. The CPU 51 performs object recognition processing on the determined image and outputs the recognition result.

[0038] The CPU 51 performs tracking processing using the acquired image. If there is a recognition result from the object recognition processing, the CPU 51 obtains the position of the object by tracking processing using an image related to the recognition result, an image temporally subsequent to the image, and the recognition result. The CPU 51 outputs the obtained position of the object as the object recognition result. If there is no recognition result from the object recognition processing but there is a result from the previous tracking processing, the CPU 51 obtains the position of the object by tracking processing using an image used in the previous tracking processing, an image temporally subsequent to the image, and the result from the previous tracking processing. The CPU 51 outputs the obtained position of the object as the object recognition result.

[0039] The computer program 531 may be stored in the HDD 53 by the computer 50 reading the computer program 531 from a recording medium, such as an optical disk, such as a CD-ROM (Compact Disc Read Only Memory), or a USB (Universal Serial Bus) memory, that stores the computer program 531. Alternatively, the computer 50 may store the computer program 531 in the HDD 53 by downloading the computer program 531 via a network, such as the Internet.

[0040] According to the computer program 531, it is possible to appropriately recognize an object while improving responsiveness in object recognition, similarly to the object recognition device 100 in the above-described embodiment. According to the computer program 531, it is possible to relatively easily realize the object recognition device 100 in the above-described embodiment.

[0041] Various aspects of the invention derived from the above-described embodiment will be described below.

[0042] An object recognition device according to one aspect of the invention includes a recognition means for performing a recognition process on a portion of a plurality of temporally consecutive images sequentially acquired in a first period, the portion being subjected to the recognition process in a second period longer than the first period, to recognize an object, and a tracking means for performing a tracking process to track the object using a first image that has been subjected to the recognition process and an image subsequent to the first image, among the plurality of images. In the above-described embodiment, the "recognizer 120" corresponds to an example of the "recognition means", and the "tracker 130" corresponds to an example of the "tracking means". The "object recognition process" in the above-described embodiment corresponds to an example of the "recognition process".

[0043] In the object recognition device, when the recognition result of the object by the recognition means is updated, the tracking means may perform the tracking process using a second image, among the plurality of images, on which the recognition process has been performed and an image subsequent to the second image, and when the recognition result of the object by the recognition means is not updated, the tracking means may perform the tracking process using a third image, among the plurality of images, that was used in the previous tracking process and an image subsequent to the third image.

[0044] An object recognition method according to one embodiment of the invention is an object recognition method in an object recognition device equipped with a recognition means and a tracking means, and includes a recognition step in which the recognition means recognizes an object by performing recognition processing on a portion of a plurality of temporally consecutive images acquired sequentially in a first period, the portion being subjected to the recognition processing in a second period longer than the first period, and a tracking step in which the tracking means performs tracking processing to track the object using a first image that has been subjected to the recognition processing and an image subsequent to the first image among the plurality of images.

[0045] A computer program according to one embodiment of the invention causes a computer provided in an object recognition device to function as: recognition means for recognizing an object by performing recognition processing on a portion of a plurality of temporally consecutive images acquired sequentially in a first period, the second period being longer than the first period; and tracking means for performing tracking processing to track the object using a first image that has been subjected to the recognition processing and an image subsequent to the first image among the plurality of images.

[0046] The present invention is not limited to the above-described embodiments, but can be modified as appropriate within the scope that does not contradict the gist or concept of the invention that can be read from the claims and the entire specification, and the object recognition device, method, and computer program that involve such modifications are also included in the technical scope of the present invention. [Explanation of symbols]

[0047] 1...vehicle, 10...camera, 100...object recognition device, 110...image assigner, 120...recognizer, 130...tracker, 140...image holder

Claims

1. An object recognition device, a recognition unit that recognizes an object by performing a recognition process on a part of a plurality of temporally consecutive images that are sequentially acquired in a first period, the second period being longer than the first period; the recognition means performs the recognition process on a first image among the plurality of images; the recognition means performs the recognition process on a second image acquired after a first recognition result is obtained by the recognition process on the first image, The object recognition device further includes a tracking unit that performs a tracking process to track the object using the first recognition result, the first image, and an image subsequent to the first image among the plurality of images, before a second recognition result is obtained by the recognition process on the second image, the tracking means tracks the object using the first recognition result, the first image, the second image, and one or more images between the first image and the second image, thereby determining a position of the object at a time when the second image was acquired; Before a second recognition result is obtained by the recognition process for the second image, the tracking means determines the position of the object at the time when the third image is acquired by tracking the object using the determined position of the object at the time when the second image is acquired, the second image, and a third image following the second image.

1. An object recognition device comprising:

2. An object recognition method in an object recognition device including a recognition means and a tracking means, the recognition means includes a recognition step of performing a recognition process on a part of a plurality of temporally consecutive images sequentially acquired in a first period, the second period being longer than the first period, to recognize an object; In the recognition step, the recognition means performs the recognition process on a first image among the plurality of images, In the recognition step, the recognition means performs the recognition process on a second image acquired after a first recognition result is obtained by the recognition process on the first image, The object recognition method further includes a tracking step in which the tracking means performs a tracking process to track the object using the first recognition result, the first image, and an image subsequent to the first image among the plurality of images; in the tracking step, before a second recognition result is obtained by the recognition process on the second image, the tracking means tracks the object using the first recognition result, the first image, the second image, and one or more images between the first image and the second image, thereby determining a position of the object at a time when the second image was acquired; In the tracking step, before a second recognition result is obtained by the recognition process on the second image, the tracking means tracks the object using the determined position of the object at the time when the second image was acquired, the second image, and a third image following the second image, thereby determining the position of the object at the time when the third image was acquired.

13. An object recognition method comprising:

3. A computer program that causes a computer included in an object recognition device to function as a recognition unit and a tracking unit, the recognition means performs a recognition process on a part of a plurality of temporally consecutive images sequentially acquired in a first period, the second period being longer than the first period, to recognize an object; the recognition means performs the recognition process on a first image among the plurality of images; the recognition means performs the recognition process on a second image acquired after a first recognition result is obtained by the recognition process on the first image, the tracking means performs a tracking process for tracking the object by using the first recognition result, the first image, and an image subsequent to the first image among the plurality of images; before a second recognition result is obtained by the recognition process on the second image, the tracking means tracks the object using the first recognition result, the first image, the second image, and one or more images between the first image and the second image, thereby determining a position of the object at a time when the second image was acquired; Before a second recognition result is obtained by the recognition process for the second image, the tracking means determines the position of the object at the time when the third image is acquired by tracking the object using the determined position of the object at the time when the second image is acquired, the second image, and a third image following the second image. A computer program comprising:

Images