Object recognition device, object recognition method, and program

Abstract

The present disclosure makes it possible to provide, for example, an object recognition device capable of improving the processing speed of object recognition even in various situations. An object recognition device according to the present disclosure is provided with: a video acquisition means that acquires a video captured from a vehicle; a first recognition means that applies the image of each frame of the video to a first recognition model to extract one or more object-like regions; a second recognition means that applies the image of each frame of the video to a second recognition model having higher accuracy than the first recognition model to recognize one or more objects; and a process switching means that applies the image of each frame of the video to the first recognition model to extract an object-like region, if a prescribed condition regarding an occupancy rate indicating the proportion of one or more object regions in the image is satisfied, and further applies the extracted object-like region to the second recognition model to recognize an object.

Description

Object recognition device, object recognition method, and program

[0001] The present disclosure relates to an object recognition device, an object recognition method, and a program.

[0002] Patent Document 1 discloses an information processing device that includes an object inference means that includes first-order inference and second-order inference.

[0003] By performing object inference on the inference target data in two stages, the information processing device can reduce the amount of calculation and shorten the processing time compared to performing object inference using a uniformly high-precision model (i.e., secondary inference).

[0004] International Publication No. 2023 / 058082

[0005] However, in the above-described information processing device, an improvement in processing speed may not be expected in various situations. An object of the present disclosure is to provide an object recognition device or the like that can improve the processing speed of object recognition in various situations.

[0006] The object recognition device according to the present disclosure comprises a video acquisition means for acquiring video captured by a vehicle; a first recognition means for applying an image of each frame of the video to a first recognition model to extract one or more object-like regions; a second recognition means for applying an image of each frame of the video to a second recognition model that is more accurate than the first recognition model to recognize one or more objects; and a processing switching means for, when a predetermined condition regarding an occupancy rate indicating the proportion of one or more object regions in the image is satisfied, applying the image of each frame of the video to the first recognition model to extract object-like regions, and further applying the extracted object-like regions to the second recognition model to recognize the object.

[0007] The object recognition method according to the present disclosure acquires a video captured by a vehicle, applies the images of each frame of the video to a first recognition model to extract one or more object-like regions, applies the images of each frame of the video to a second recognition model that is more accurate than the first recognition model to recognize one or more objects, and if a predetermined condition regarding an occupancy rate indicating the proportion of one or more object regions in the image is satisfied, applies the images of each frame of the video to the first recognition model to extract object-like regions, and further applies the extracted object-like regions to the second recognition model to recognize the object.

[0008] The program disclosed herein causes a computer to acquire a video captured by a vehicle, apply images of each frame of the video to a first recognition model to extract one or more object-like regions, apply images of each frame of the video to a second recognition model that is more accurate than the first recognition model to recognize one or more objects, and if a predetermined condition regarding an occupancy rate indicating the proportion of one or more object regions in the image is satisfied, apply images of each frame of the video to the first recognition model to extract object-like regions, and further apply the extracted object-like regions to the second recognition model to recognize the object.

[0009] According to the present disclosure, it is possible to provide an object recognition device and the like that can improve the processing speed of object recognition in various situations.

[0010] FIG. 1 is a block diagram illustrating an example configuration of an object recognition device according to an embodiment. FIG. 2 is a flowchart illustrating an object recognition method according to an embodiment. FIG. 3 is a block diagram illustrating an example configuration of a learning device according to an embodiment. FIG. 4 is a flowchart illustrating a learning method according to an embodiment. FIG. 5 is a diagram illustrating a method for accelerating object recognition processing. FIG. 6 is a block diagram illustrating an example configuration of an object recognition device according to another embodiment. FIG. 7 is a flowchart illustrating a flow of switching from non-application to application of incremental object recognition technology according to another embodiment. FIG. 8 is a flowchart illustrating a flow of switching from non-application to application of incremental object recognition technology according to another embodiment. FIG. 9 is a flowchart illustrating a flow of switching from application to non-application of incremental object recognition technology according to another embodiment. FIG. 10 is a block diagram illustrating an example hardware configuration of an object recognition device, etc.

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the drawings, the same elements are designated by the same reference numerals, and redundant explanations will be omitted as necessary.

[0012] When referring to one embodiment below, it means that the present invention can be applied to any one of the embodiments described below or a combination of two or more embodiments, and is not limited to a specific embodiment.

[0013] 1 is a block diagram illustrating an example of the configuration of an object recognition device according to an embodiment. The object recognition device 100 is implemented by a computer including a processor and a memory. The object recognition device 100 includes a video image acquisition unit 101, a first recognition unit 102, a second recognition unit 103, and a processing switching unit 108.

[0014] The video image acquisition unit 101, also referred to as a video image acquisition means, acquires video images captured by a vehicle. At least a portion of the video image data may include an object. Here, the object may be, for example, an object that may affect the running of the vehicle, such as another vehicle, a pedestrian, a bicycle, or a traffic light.

[0015] The first recognition unit 102, also referred to as the first recognition means, applies a first recognition model to the image of each frame of a video to extract one or more object-like regions. The area excluding the one or more object-like regions extracted from the image becomes the background region. The first recognition model is a lighter model than the second recognition model described below, and is capable of high-speed processing, but tends to have lower recognition accuracy.

[0016] The second recognition unit 103, also called a second recognition means, applies a second recognition model, which has higher accuracy than the first recognition model, to each frame of the video to recognize one or more objects. A region of the image that surrounds the recognized one or more objects is also called an object region. The object region may be, for example, the coordinates of the four vertices of a rectangular region in which the target object is captured (bounding box (BB)).

[0017] The processing switching unit 108, also referred to as a processing switching means, applies a progressive object recognition technique, which is an object recognition method including first-order inference and second-order inference, when a predetermined condition regarding an occupancy rate indicating the proportion of one or more object regions in an image is satisfied. That is, the processing switching unit 108 applies a first recognition model to an image of each frame of a video to extract an object-like region, and then further applies the extracted object-like region to a second recognition model to recognize the object.

[0018] 2 is a flowchart illustrating an object recognition method according to one embodiment. The video acquisition unit 101 acquires video captured by a vehicle (S101). The first recognition unit 102 applies a first recognition model to the image of each frame of the video to extract one or more object-like regions (S102). The second recognition unit 103 applies a second recognition model, which has higher accuracy than the first recognition model, to the image of each frame of the video to recognize one or more objects (S103).

[0019] When a predetermined condition regarding the occupancy rate, which indicates the proportion of one or more object areas in the image, is met, the processing switching unit 108 applies the image of each frame of the video to a first recognition model to extract areas that resemble objects, and further applies the extracted object-like areas to a second recognition model to recognize the object (S104).

[0020] The object recognition device and method described above can improve processing speed in a variety of situations by applying incremental object recognition technology, which is an object recognition method that includes first-order inference and second-order inference, when certain conditions regarding occupancy are met.

[0021] 3 is a block diagram illustrating an example configuration of a learning device according to an embodiment. The learning device 10 is realized by a computer having a memory and a processor. The learning device 10 includes a data storage unit 50, a data generation unit 60, a model generation unit 70, and a model storage unit 80. Some or all of the functions of the learning device 10 may be included in the object recognition device described above.

[0022] 4 is a flowchart illustrating a learning method according to one embodiment. The learning device 10 starts generating a trained model when a predetermined condition is met. For example, the learning device 10 starts generating a trained model when an instruction from an operator is received. In this case, at the start of operation, the learning device 10 may acquire parameters required for generating the trained model from the operator.

[0023] The parameters required for generating a trained model are arbitrary. For example, if the data storage unit 50 stores multiple training datasets, the parameters are information indicating which training dataset to use. Alternatively, the parameters may be parameters related to the machine learning process in the model generation unit 70. Note that the learning device 10 may acquire other information in addition to the parameters. For example, the data generation unit 60 may acquire at least a portion of the training dataset from an operator. In this case, the data generation unit 60 stores the acquired training dataset in the data storage unit 50.

[0024] The data generation unit 60 generates a primary inference training dataset using the original training dataset stored in the data storage unit 50 (S11). For example, the data generation unit 60 generates a primary inference training dataset using supervised answer data included in the original training dataset. The data generation unit 60 then stores the primary inference training dataset in the data storage unit 50. The model generation unit 70 performs machine learning in a predetermined model using the primary inference training dataset to generate a trained model for primary inference (step S12). The model generation unit 70 then stores the trained model for primary inference in the model storage unit 80 as the first recognition model described above.

[0025] The data generation unit 60 generates a secondary inference training dataset using the original training dataset stored in the data storage unit 50 (step S13). For example, the data generation unit 60 may generate aggregated data using the supervised data of the original training dataset, and generate a secondary inference training dataset including the supervised data and the aggregated data. The data generation unit 60 then stores the generated secondary inference training dataset in the data storage unit 50. The model generation unit 70 performs machine learning in a predetermined model using the secondary inference training dataset to generate a trained model for secondary inference (step S14). The model generation unit 70 then stores the trained model for secondary inference in the model storage unit 80 as the second recognition model described above. Note that after generating the trained model, the learning device 10 may notify the operator of the execution results.

[0026] Other Embodiments One of the main tasks using machine learning is object detection in an image. The object detection task is a task for generating a list of pairs of positions and classes (types) of target objects present in an image. In recent years, object detection tasks using deep learning, in particular, have become widely used in machine learning. In the object detection task, a group of training images and information about the target object in each image are provided as ground truth data in the learning phase of machine learning. The information about the target object is selected in accordance with the specifications of the object detection task. For example, the information about the target object includes the coordinates of the four vertices of a rectangular area in which the target object is captured (bounding box (BB))) and the class of the target object. For example, the information about the target object includes the coordinates of the four vertices of a rectangular area in which the target object is captured (bounding box (BB))) and the class of the target object. Note that the following description also uses BB and class as an example of information about the target object. The object detection task then generates a trained model as a result of machine learning using, for example, deep learning, using a group of training images and information about the target objects. In the detection phase, the object detection task applies the trained model to images containing the target objects to infer the target objects contained in the images, and outputs a pair of BB and class for each target object contained in the image. Note that the object detection task may also output an evaluation result for the object detection results along with the BB and class. While confidence is often used as the evaluation result, other values ​​may also be used.

[0027] For example, a person and vehicle surveillance system can be built using an object detection task. For example, a system user inputs images from a surveillance camera into the object detection task. The object detection task detects the positions and classes of people and vehicles in the images. The user can then use the positions and classes of people and vehicles detected by the object detection task to determine the people and vehicles appearing in the surveillance camera images. The object detection task may also be used in combination with other recognition processes. For example, a license plate recognition system can be built by combining an object detection task that detects vehicle license plates with an optical character recognition (OCR) process that recognizes characters on the detected license plates.

[0028] In a system for detecting objects in images captured by a surveillance camera, the surveillance camera or peripheral devices around the surveillance camera, i.e., the edge, are assumed to be configured to perform inference processing for the object detection task. However, the edge is often constrained by installation space, cooling, power, and other factors, and only limited computing resources are available. Even in such an edge environment, processing such as object detection requires a certain level of high processing speed.

[0029] However, inference processing using machine learning, particularly object detection and pose estimation tasks, is computationally intensive and time-consuming. As a result, object detection and pose estimation tasks generally tend to have low throughput performance. Therefore, it is desirable to improve the throughput of inference processing in object detection and pose estimation tasks.

[0030] FIG. 5 is a diagram illustrating a method for accelerating object recognition processing. First, the information processing device acquires, as input images, images of each frame of a video captured by an imaging unit mounted on a vehicle. In a first stage, the information processing device applies the input image to a first recognition model to extract a background region and one or more object-like regions. Here, the first recognition model is a trained model used for primary inference, and is smaller and lighter than the trained model used in secondary inference, which will be described later. In FIG. 5, the object-like region is a region surrounding an object recognized using this first recognition model.

[0031] Next, the recognition result 1 of the primary inference is sent to a high-precision model for secondary inference. In the second stage, the information processing device recognizes the object using the high-precision model based on the recognition result of the primary inference. That is, the high-precision model is applied to an area of ​​the image that is likely to be an object, and object recognition is performed. Note that such two-stage object recognition may also be called an incremental object recognition technology.

[0032] As described above, by performing object recognition in two stages, combining primary object inference using a small model and secondary object inference using a high-precision model, the amount of calculation can be reduced and processing can be sped up compared to performing object inference using a high-precision model uniformly on the entire image.

[0033] However, the inventors have found that an improvement in processing speed may not be expected in various situations. Specifically, when the result of primary object inference shows that the area occupied by one or more objects is large relative to the entire image, i.e., when the area occupied by the background area relative to the entire image is small, the above-mentioned two-stage object recognition may result in a slow processing speed.

[0034] The various situations in which improvement of processing speed is not expected include when the area of ​​one or more objects occupies a large portion of the entire image, i.e., when the background area occupies a small portion of the entire image. Here, the ratio of the area of ​​one or more objects to the entire image is referred to as the occupancy rate. When this occupancy rate is high, even if incremental object recognition technology is used, improvement of processing speed is not expected.

[0035] If a predetermined condition regarding the occupancy rate is satisfied, the occupancy rate is assumed to be low, and therefore, the progressive object recognition technology is applied. Specifically, for example, when a vehicle is traveling on a predetermined type of road, such as a highway or a bypass road, or when the vehicle is traveling at a predetermined speed or above, the inter-vehicle distance is often large, and therefore, the proportion of the background area is likely to be high. Therefore, in this case, the predetermined condition regarding the occupancy rate is satisfied, i.e., the occupancy rate is assumed to be low, and therefore, the progressive object recognition technology is applied. Note that, in this example, the application to a highway is used as an example of a case where the predetermined condition regarding the occupancy rate is satisfied, but this is not limiting. For example, roads in rural areas with few vehicles and pedestrians may be assumed to have a low occupancy rate, and therefore, the progressive object recognition technology may be applied to such roads, which are included in the above-mentioned predetermined type of road.

[0036] Conversely, when a vehicle is traveling or stopped at a speed lower than a predetermined speed on a certain type of road, i.e., when the road is congested on a highway or bypass road, it is considered that the area of ​​one or more objects is large (high occupancy), and in this case, the progressive object recognition technology is not applied.

[0037] If the vehicle is not on a road of a predetermined type or is congested on a road of a predetermined type, the object is recognized by a high-precision object recognition model, and the occupancy rate is calculated. If the calculated occupancy rate is lower than a threshold, the progressive object recognition technology is applied.

[0038] In some embodiments, the primary inference results may be prioritized according to the reliability of the object recognition, and those with low priorities may not be sent to the secondary inference. This allows for a further reduction in the amount of calculations and processing time by not recognizing low-priority objects using a high-precision object recognition model.

[0039] 6 is a block diagram illustrating an example configuration of an object recognition device according to another embodiment. The object recognition device 100a includes a video image acquisition unit 101a, a first recognition unit 102a, a second recognition unit 103a, a position information acquisition unit 104a, a traveling speed acquisition unit 105a, an occupancy rate calculation unit 106a, a reliability calculation unit 107a, and a processing switching unit 108a. The object recognition device 100a is realized by a computer including a processor, a memory, and the like. The object recognition device 100a can be mainly used by being mounted on a vehicle.

[0040] The video image acquisition unit 101a is also called a video image acquisition means and acquires video images captured by a vehicle. An imaging unit mounted on the vehicle captures images of the area ahead in the direction in which the vehicle is traveling.

[0041] The first recognition unit 102a, also referred to as a first recognition means, applies a first recognition model to an image of each frame of the acquired video to extract a background region and one or more object-like regions. The first recognition unit 102a may apply the first recognition model to extract one or more object-like regions, thereby regarding other regions as background regions. The first recognition model is a trained model used for primary inference, and is smaller and lighter than the trained model used for secondary inference described below.

[0042] The second recognition unit 103a is also called a second recognition means, and recognizes one or more objects by applying an image of each frame of a video to a second recognition model that is more accurate than the first recognition model. The second recognition model is a trained model for secondary inference and is more accurate than the first recognition model. The second recognition model may also be called a recognition model for secondary inference, a second object recognition model, or an object recognition engine.

[0043] The processing switching unit 108a, also referred to as a processing switching means, applies a progressive object recognition technique when a predetermined condition regarding an occupancy rate, which indicates the proportion of one or more objects in an image, is satisfied. That is, when the predetermined condition is satisfied, the processing switching unit 108a applies a first recognition model to the image of each frame of the video to extract an object-like region, and then further applies a second recognition model to the extracted object-like region to recognize the object.

[0044] The location information acquisition unit 104a, also referred to as location information acquisition means, acquires vehicle location information from a receiver of a satellite positioning system (e.g., a Global Positioning System (GPS)) mounted on the vehicle. For example, an image captured by an imaging unit mounted on a vehicle traveling on a highway at a predetermined speed or above often has a large background area and a small object area (i.e., a low occupancy). Therefore, the location information acquisition unit 104a sends a signal indicating the vehicle location information to the processing switching unit 108a, and the processing switching unit 108a can apply incremental object recognition technology.

[0045] The traveling speed acquisition unit 105a, also referred to as a traveling speed acquisition means, acquires the vehicle's speed from a vehicle speedometer mounted on the vehicle. For example, images captured by an imaging unit mounted on a vehicle traveling on a highway at less than a predetermined speed often capture many vehicles stuck in traffic, with a small background area and a large object area (i.e., a high occupancy rate). Therefore, the traveling speed acquisition unit 105a sends a signal indicating the vehicle's speed to the processing switching unit 108a, which may forgo application of the incremental object recognition technology. In this case, the processing switching unit 108a may apply the second recognition unit 103a to apply a second recognition model, which has higher accuracy than the first recognition model, to images of each frame of the video to recognize one or more objects.

[0046] It is also possible to determine whether the vehicle is traveling on a highway from the traveling speed acquired by the traveling speed acquisition unit 105a. For example, if the acquired traveling speed is 80 km / h or higher, it may be determined that the vehicle is traveling on a highway. The traveling speed of the vehicle may also be determined from the amount of change in the vehicle position acquired by the position information acquisition unit 104a. In other words, the object recognition device 100a may have only one of the position information acquisition unit 104a and the traveling speed acquisition unit 105a.

[0047] In addition, when determining whether a traffic jam is occurring on a highway, the distance between vehicles obtained from a camera, LiDAR, or radar may also be used in combination.

[0048] The occupancy calculation unit 106a, also referred to as an occupancy calculation means, calculates an occupancy rate indicating the proportion of one or more object regions in an image captured by an imaging unit mounted on the vehicle. As described above, when the vehicle is traveling on a highway at a predetermined speed or higher, the occupancy rate is not calculated, and when the occupancy rate is assumed to be low, the progressive object recognition technology is applied. However, when the vehicle is traveling on an ordinary road, for example, whether or not to apply the progressive object recognition technology varies depending on the situation. Therefore, the occupancy calculation unit 106a may calculate the occupancy rate, and when the calculated occupancy rate is lower than a threshold, the processing switching unit 108a may apply the progressive object recognition technology.

[0049] The reliability calculation unit 107a, also called a reliability calculation means, calculates the reliability of one or more objects recognized by the first recognition unit 102a or the second recognition unit 103a. The reliability is calculated as a numerical value (for example, a percentage), so that objects with a reliability below a threshold can be excluded.

[0050] The process switching unit 108a switches between various object recognition processes, including the incremental object recognition technique and the high-precision object recognition, based on the vehicle's position information, traveling speed information, occupancy rate, and reliability, thereby improving the object recognition processing speed in various situations.

[0051] FIG. 7 is a flowchart illustrating a process for switching from non-application to application of incremental object recognition technology according to another embodiment. A vehicle travels while capturing images of the area ahead in the traveling direction using an onboard imaging unit, and recognizing obstacles (objects) in the captured images using the object recognition device 100. Specifically, the video acquisition unit 101a acquires video captured by the vehicle. The first recognition unit 102a applies the images of each frame of the acquired video to a first recognition model to extract a background region and one or more object-like regions. The second recognition unit 103a applies the images of each frame of the video to a second recognition model with higher accuracy than the first recognition model to recognize one or more objects.

[0052] First, the location information acquisition unit 104a acquires vehicle location information from a receiver of a satellite positioning system mounted on the vehicle. If the vehicle is traveling on a highway (YES in S501), the traveling speed acquisition unit 105a acquires the vehicle speed from a speedometer. If the vehicle speed within a certain period of time is equal to or greater than a threshold (i.e., the vehicle is not in a traffic jam) (YES in S502), the processing switching unit 108a applies an incremental object recognition technique including primary inference and secondary inference.

[0053] On the other hand, if the location information from the location information acquisition unit 104a indicates that the vehicle is not on a highway (NO in S501), or if the driving speed from the driving speed acquisition unit 105a indicates that the vehicle is stuck in a traffic jam on a highway (NO in S502), the processing switching unit 108a applies the images of each frame of the video to the second recognition model to recognize objects (S510) and acquires obstacle information that may affect the driving of the vehicle (S512).

[0054] The occupancy calculation unit 106a calculates an occupancy rate indicating the proportion of one or more object regions in an image captured by an imaging unit mounted on the vehicle (S514). If the calculated occupancy rate is less than a lower threshold Lo (YES in S516) and a certain period of time has elapsed (YES in S518), the processing switching unit 108a applies an incremental object recognition technique including first-order inference and second-order inference (S505).

[0055] If the calculated occupancy rate is equal to or greater than the lower threshold Lo (NO in S516) and if the calculated occupancy rate exceeds the upper threshold Hi (YES in S520), the processing switching unit 108a refrains from applying the gradual object recognition technology and terminates the processing.

[0056] If the calculated occupancy rate is equal to or greater than the lower threshold Lo (NO in S516) and equal to or less than the upper threshold Hi (NO in S520), the process proceeds to the process of FIG.

[0057] FIG. 8 is a flowchart illustrating a process for switching from non-application to application of the incremental object recognition technology according to another embodiment. From the recognition results of the second recognition model (S510) described above in FIG. 5, one or more obstacles (objects) in each frame of a video image whose reliability is equal to or less than a threshold x are excluded, and the occupancy rate is calculated again (S601). For example, if the recognition results show object A (reliability 80%), object B (reliability 75%), and object C (reliability 15%) in the image, object C, whose reliability is equal to or less than the threshold (50%), is excluded. Then, the occupancy rates are calculated again for object A (reliability 80%) and object B (reliability 75%) in the image. Note that these reliability values ​​are merely exemplary and are not limiting.

[0058] If the calculated occupancy rate is less than the threshold Lo (YES in S603) and a certain period of time has elapsed (YES in S605), the incremental object recognition technique is performed (S610). Specifically, the first recognition unit 102a applies the first recognition model to each frame of the acquired video as a primary inference to extract a background region and one or more object-like regions (S611). Then, from the extracted one or more object-like regions, obstacles (objects) recognized by the first recognition model with a reliability equal to or less than threshold y are excluded (S613). The second recognition unit 103a applies the extracted one or more object-like regions (excluding obstacles (objects) recognized by the first recognition model with a reliability equal to or less than threshold y) to a second recognition model with higher accuracy than the first recognition model to recognize one or more objects (S615). Note that threshold x is the recognition threshold according to the second recognition model, and threshold y is the recognition threshold according to the first recognition model, and they may be different from each other.

[0059] As described above with reference to Figures 7 and 8, in certain situations, such as when traveling on a highway at a predetermined speed or higher, the occupancy rate is not calculated and the incremental object recognition technology is applied. In situations other than these specific situations, the occupancy rate is calculated from the object recognition results, and if the occupancy rate is below a threshold value Lo, the incremental object recognition technology is applied. If the occupancy rate is above a lower threshold value Lo and below an upper threshold value Hi, objects whose reliability is below the threshold value from the object recognition results are excluded and the incremental object recognition technology is applied. This makes it possible to improve the processing speed of object recognition in various situations.

[0060] 9 is a flowchart illustrating a process for switching from application to non-application of incremental object recognition technology according to another embodiment. After applying the incremental object recognition technology, the second recognition unit 103a applies a second recognition model, which has higher accuracy than the first recognition model, to each frame of the acquired video to recognize one or more objects and acquire the recognized objects as obstacle information that may affect vehicle travel (S701, S703). The occupancy calculation unit 106a calculates an occupancy rate, which indicates the proportion of one or more obstacle (object) areas in an image captured by an imaging unit mounted on the vehicle (S705). If the calculated occupancy rate is greater than a threshold C (YES in S707) and a certain period of time has elapsed (YES in S709), the processing switching unit 108a switches from application of the incremental object recognition technology to non-application (S710).

[0061] A computer program can be stored and provided to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, and RAMs (Random Access Memory)). The program may be provided to the computer by various types of transient computer-readable media. Examples of transient computer-readable media include electrical signals, optical signals, and electromagnetic waves. The transient computer-readable media can provide the program to the computer via a wired communication path such as an electrical wire or optical fiber, or via a wireless communication path.

[0062] FIG. 10 is a block diagram showing an example of the hardware configuration of the object recognition device 100, 100a, and the learning device 10 (hereinafter referred to as the object recognition device 100, etc.). Referring to FIG. 10, the object recognition device 100, etc. includes a network interface 1201, a processor 1202, and a memory 1203. The network interface 1201 is used to communicate with other network node devices constituting a communication system. The network interface 1201 may be used for wireless communication. For example, the network interface 1201 may be used for wireless LAN communication defined in the IEEE 802.11 series or mobile communication defined in 3GPP (registered trademark) (3rd Generation Partnership Project). Alternatively, the network interface 1201 may include, for example, a network interface card (NIC) conforming to the IEEE 802.3 series.

[0063] The processor 1202 reads and executes software (computer programs) from the memory 1203 to perform the processing of the object recognition device 100 and the like described using flowcharts or sequences in the above-described embodiments. The processor 1202 may be, for example, a microprocessor, an MPU (Micro Processing Unit), a CPU (Central Processing Unit), or a GPU (Graphics Processing Unit). The processor 1202 may include multiple processors.

[0064] The memory 1203 is configured by a combination of volatile memory and non-volatile memory. The memory 1203 may include storage located remotely from the processor 1202. In this case, the processor 1202 may access the memory 1203 via an I / O interface (not shown).

[0065] 10, the memory 1203 is used to store software modules. The processor 1202 reads and executes these software modules from the memory 1203, thereby performing the processing of the object recognition device 100 described in the above embodiment, etc.

[0066] As explained using FIG. 10 , each of the processors included in the object recognition device 100 executes one or more programs including a group of instructions for causing a computer to perform the algorithm explained using the drawing.

[0067] Although the object recognition device 100 and the like have been described as a single device here, this is merely an example. The object recognition device 100 and the like may be configured by a plurality of physically separated devices. Some of the plurality of devices may be transportable devices, and the other devices may be stationary devices.

[0068] Each drawing is merely an example for describing one or more embodiments. Each drawing may not relate to only one particular embodiment, but may also relate to one or more other embodiments. As will be understood by those skilled in the art, various features or steps described with reference to any one drawing can be combined with features or steps shown in one or more other drawings to create, for example, an embodiment not explicitly shown or described. Not all features or steps shown in any one drawing are necessary to describe an exemplary embodiment, and some features or steps may be omitted. The order of steps described in any drawing may be changed as appropriate.

[0069] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes.

[0070] (Supplementary Note 1) An object recognition device comprising: a video acquisition means for acquiring a video captured by a vehicle; a first recognition means for applying an image of each frame of the video to a first recognition model and extracting one or more object-like regions; a second recognition means for applying the image of each frame of the video to a second recognition model that is more accurate than the first recognition model and recognizing one or more objects; and a processing switching means for, when a predetermined condition related to an occupancy rate indicating the proportion of one or more object regions in the image is satisfied, applying the image of each frame of the video to the first recognition model to extract an object-like region, and further applying the extracted object-like region to the second recognition model to recognize the object. (Supplementary Note 2) The object recognition device according to Supplementary Note 1, wherein the predetermined condition related to the occupancy rate is satisfied when the vehicle is traveling on a predetermined type of road. (Supplementary Note 3) The object recognition device according to Supplementary Note 1 or 2, wherein the predetermined condition related to the occupancy rate is satisfied when the vehicle is traveling at a predetermined speed or higher. (Supplementary Note 4) The object recognition device according to Supplementary Note 1, wherein a predetermined condition regarding the occupancy rate is satisfied when the occupancy rate of the object region in the image of each frame of the video continues to be less than a lower threshold for a predetermined period of time.(Supplementary Note 5) The object recognition device described in Supplementary Note 1, wherein a predetermined condition regarding the occupancy rate is satisfied when: the vehicle is not on a predetermined type of road, or the vehicle is traveling or stopped on a predetermined type of road at less than a predetermined speed; the image of each frame of the video is applied to the second recognition model to recognize an object, and the occupancy rate of the object region in the image is calculated; and if the calculated occupancy rate is equal to or greater than a lower threshold and equal to or less than an upper threshold, the occupancy rate is calculated again, excluding objects whose reliability recognized by the second recognition model is less than a threshold; and the occupancy rate is calculated again, excluding the recalculated occupancy rate and a predetermined time has elapsed; the processing switching means applies the image of each frame of the video to the first recognition model to extract one or more object-like regions, excluding from the extracted one or more object-like regions those whose reliability recognized by the first recognition model is less than a threshold, and further applying the remaining one or more object-like regions to the second recognition model to recognize the object. (Supplementary Note 6) The object recognition device according to Supplementary Note 1, wherein the processing switching means switches to applying images of each frame of the video to the second recognition model to recognize objects when the predetermined condition related to the occupancy rate is not satisfied. (Supplementary Note 7) An object recognition method comprising: acquiring video captured by a vehicle; applying images of each frame of the video to a first recognition model to extract one or more object-like regions; applying images of each frame of the video to a second recognition model that is more accurate than the first recognition model to recognize one or more objects; and when a predetermined condition related to occupancy rate indicating the proportion of one or more object regions in the image is satisfied, applying images of each frame of the video to the first recognition model to extract object-like regions, and further applying the extracted object-like regions to the second recognition model to recognize objects. (Supplementary Note 8) The object recognition method according to Supplementary Note 7, wherein a case where the predetermined condition related to occupancy rate is satisfied is when the vehicle is traveling on a predetermined type of road. (Supplementary Note 9) The object recognition method according to Supplementary Note 7 or 8, wherein a predetermined condition regarding the occupancy rate is satisfied when the vehicle is traveling at a predetermined speed or faster.(Supplementary Note 10) The object recognition method according to Supplementary Note 7, wherein a predetermined condition regarding the occupancy rate is satisfied when the occupancy rate of the object region in the image of each frame of the video continues to be less than a lower threshold for a predetermined period of time. (Supplementary Note 11) The object recognition method according to Supplementary Note 7, wherein a predetermined condition regarding the occupancy rate is satisfied when: the vehicle is not on a road of a predetermined type, or the vehicle is traveling or stopped on a road of a predetermined type at less than a predetermined speed; applying images of each frame of the video to the second recognition model to recognize an object and calculating the occupancy rate of the object region in the image; and if the calculated occupancy rate is equal to or greater than a lower threshold and equal to or less than an upper threshold, excluding objects recognized by the second recognition model whose reliability is equal to or less than a threshold and calculating the occupancy rate again; and if the recalculated occupancy rate is less than the threshold and a predetermined time has elapsed, applying images of each frame of the video to the first recognition model to extract one or more object-like regions, excluding from the extracted one or more object-like regions those whose reliability of the object recognized by the first recognition model is equal to or less than a threshold, and further applying the remaining one or more object-like regions to the second recognition model to recognize the object. (Supplementary Note 12) The object recognition method according to Supplementary Note 7, wherein, when the predetermined condition regarding the occupancy rate is not satisfied, the image of each frame of the video is applied to the second recognition model to recognize the object. (Supplementary Note 13) A program that causes a computer to execute the following: acquire a video captured by a vehicle; apply the image of each frame of the video to a first recognition model to extract one or more object-like regions; recognize one or more objects by applying the image of each frame of the video to a second recognition model that is more accurate than the first recognition model; and, when a predetermined condition regarding the occupancy rate indicating the proportion of one or more object regions in the image is satisfied, apply the image of each frame of the video to the first recognition model to extract object-like regions, and further apply the extracted object-like regions to the second recognition model to recognize the object. (Supplementary Note 14) The program according to Supplementary Note 13, wherein the predetermined condition regarding the occupancy rate is satisfied when the vehicle is traveling on a predetermined type of road.(Supplementary Note 15) The program according to Supplementary Note 13 or 14, wherein the predetermined condition regarding the occupancy rate is satisfied when the vehicle is traveling at a predetermined speed or higher. (Supplementary Note 16) The program according to Supplementary Note 13, wherein the predetermined condition regarding the occupancy rate is satisfied when a state in which the occupancy rate of the object region in images of each frame of the video is less than a lower limit threshold continues for a predetermined time. (Supplementary Note 17) The program described in Supplementary Note 13, wherein when a predetermined condition regarding the occupancy rate is satisfied: when the vehicle is not on a predetermined type of road, or when the vehicle is traveling or stopped on a predetermined type of road at less than a predetermined speed, the program applies images of each frame of the video to the second recognition model to recognize an object and calculates the occupancy rate of the object region in the image, and when the calculated occupancy rate is greater than or equal to a lower threshold and less than an upper threshold, excludes objects recognized by the second recognition model whose reliability is less than a threshold and calculates the occupancy rate again; and when the recalculated occupancy rate is less than the threshold and a predetermined time has passed, the program applies images of each frame of the video to the first recognition model to extract one or more object-like regions, excludes from the extracted one or more object-like regions those whose reliability of the object recognized by the first recognition model is less than a threshold, and further applies the remaining one or more object-like regions to the second recognition model to recognize the object. (Supplementary Note 18) The program according to Supplementary Note 13, wherein, when the predetermined condition regarding the occupancy rate is not satisfied, the image of each frame of the video is applied to the second recognition model to switch to recognizing the object.

[0071] 10 Learning device 50 Data storage unit 60 Data generation unit 70 Model generation unit 75 Reliability 80 Model storage unit 100, 100a Object recognition device 101, 101a Video image acquisition unit 102, 102a First recognition unit 103, 103a Second recognition unit 104a Position information acquisition unit 105a Travel speed acquisition unit 106a Occupancy calculation unit 107a Reliability calculation unit 108, 108a Processing switching unit

Claims

1. An object recognition device comprising: a video acquisition means for acquiring video captured by a vehicle; a first recognition means for applying the image of each frame of the video to a first recognition model and extracting one or more object-like regions; a second recognition means for applying the image of each frame of the video to a second recognition model which is more accurate than the first recognition model and recognizing one or more objects; and a processing switching means for, when a predetermined condition regarding an occupancy rate indicating the proportion of one or more object regions in the image is satisfied, applying the image of each frame of the video to the first recognition model to extract object-like regions, and further applying the extracted object-like regions to the second recognition model to recognize the object.

2. The object recognition device according to claim 1, wherein a predetermined condition regarding the occupancy rate is satisfied when the vehicle is traveling on a predetermined type of road.

3. The object recognition device according to claim 1 or 2, wherein the predetermined condition regarding the occupancy rate is satisfied when the vehicle is traveling at a predetermined speed or faster.

4. The object recognition device according to claim 1, wherein the predetermined condition regarding the occupancy rate is satisfied when the occupancy rate of the object region in the image of each frame of the video continues to be less than a lower threshold for a predetermined period of time.

5. The object recognition device of claim 1, wherein the process switching means: applies the images of each frame of the video to the first recognition model to extract one or more object-like regions; excludes from the extracted one or more object-like regions those for which the reliability of the object recognized by the first recognition model is below the threshold; and further applies the remaining one or more object-like regions to the second recognition model to recognize objects; and if the calculated occupancy rate is equal to or greater than a lower threshold and equal to or less than an upper threshold, the process switching means: applies the images of each frame of the video to the first recognition model to extract one or more object-like regions; excludes from the extracted one or more object-like regions those for which the reliability of the object recognized by the first recognition model is below the threshold; and further applies the remaining one or more object-like regions to the second recognition model to recognize objects.

6. The object recognition device according to claim 1, wherein the processing switching means switches to applying the second recognition model to the images of each frame of the video to recognize the object when the predetermined condition regarding the occupancy rate is not satisfied.

7. An object recognition method comprising: acquiring a video captured by a vehicle; applying the images of each frame of the video to a first recognition model to extract one or more object-like regions; applying the images of each frame of the video to a second recognition model that is more accurate than the first recognition model to recognize one or more objects; and, if a predetermined condition regarding occupancy indicating the proportion of one or more object regions in the image is satisfied, applying the images of each frame of the video to the first recognition model to extract object-like regions; and further applying the extracted object-like regions to the second recognition model to recognize the object.

8. The object recognition method according to claim 7, wherein a predetermined condition regarding the occupancy rate is satisfied when the vehicle is traveling on a predetermined type of road.

9. The object recognition method according to claim 7 or 8, wherein a predetermined condition regarding the occupancy rate is satisfied when the vehicle is traveling at a predetermined speed or faster.

10. The object recognition method described in claim 7, wherein a predetermined condition regarding the occupancy rate is satisfied when the occupancy rate of the object area in the image of each frame of the video image remains below a lower threshold for a predetermined period of time.

11. The object recognition method of claim 7, wherein a predetermined condition regarding the occupancy rate is satisfied: when the vehicle is not on a predetermined type of road, or when the vehicle is traveling or stopped on a predetermined type of road at less than a predetermined speed, the image of each frame of the video is applied to the second recognition model to recognize an object, and the occupancy rate of the object region in the image is calculated; and when the calculated occupancy rate is equal to or greater than a lower threshold and equal to or less than an upper threshold, objects recognized by the second recognition model whose reliability is equal to or less than the threshold are excluded and the occupancy rate is calculated again; and when the recalculated occupancy rate is less than the threshold and a predetermined time has passed, the image of each frame of the video is applied to the first recognition model to extract one or more object-like regions, and from the extracted one or more object-like regions, objects recognized by the first recognition model whose reliability is equal to or less than the threshold are excluded and the remaining one or more object-like regions are further applied to the second recognition model to recognize the object.

12. The object recognition method according to claim 7, wherein if the predetermined condition regarding the occupancy rate is not satisfied, the image of each frame of the video is switched to the second recognition model to recognize the object.

13. A program that causes a computer to perform the following steps: acquire a video captured by a vehicle; apply the images of each frame of the video to a first recognition model to extract one or more object-like regions; apply the images of each frame of the video to a second recognition model that is more accurate than the first recognition model to recognize one or more objects; and, if a predetermined condition regarding occupancy indicating the proportion of one or more object regions in the image is met, apply the images of each frame of the video to the first recognition model to extract object-like regions, and further apply the extracted object-like regions to the second recognition model to recognize the object.

14. The program according to claim 13, wherein the predetermined condition regarding the occupancy rate is satisfied when the vehicle is traveling on a predetermined type of road.

15. The program according to claim 13 or 14, wherein a predetermined condition regarding the occupancy rate is satisfied when the vehicle is traveling at a predetermined speed or faster.

16. The program according to claim 13, wherein the predetermined condition regarding the occupancy rate is satisfied when the occupancy rate of the object region in the image of each frame of the video continues to be less than a lower threshold for a predetermined period of time.

17. The program described in claim 13, wherein, when a predetermined condition regarding the occupancy rate is met: when the vehicle is not on a predetermined type of road, or when the vehicle is traveling or stopped on a predetermined type of road at less than a predetermined speed, the program applies the images of each frame of the video to the second recognition model to recognize an object and calculates the occupancy rate of the object region in the image, and when the calculated occupancy rate is equal to or greater than a lower threshold and equal to or less than an upper threshold, excludes objects recognized by the second recognition model whose reliability is below the threshold and calculates the occupancy rate again; and when the recalculated occupancy rate is below the threshold and a predetermined time has passed, the program applies the images of each frame of the video to the first recognition model to extract one or more object-like regions, excludes from the extracted one or more object-like regions those whose reliability recognized by the first recognition model is below the threshold, and further applies the remaining one or more object-like regions to the second recognition model to recognize the object.

18. The program according to claim 13, wherein if the predetermined condition regarding the occupancy rate is not satisfied, the image of each frame of the video is applied to the second recognition model to switch to recognizing the object.

Images