Method and device for identifying model by using traffic light

Abstract

A method of detecting traffic lights using an object recognition model configured to recognize traffic lights is disclosed. The method includes: obtaining an input image from a camera included in a vehicle, the input image being among frames captured by the camera including a previous frame; estimating a first region of interest (RoI) for the input image based on a priori information about the traffic light object; determining a second RoI based on the first RoI and based on a detection result of the previous frame, the detection result corresponding to a recognition result of recognizing a traffic light in the previous frame by an object recognition model; and identifying traffic lights in the input image by an object identification model, where the identification is based on the input image and the second RoI.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of Korean Patent Application No. 10-2023-0163309 filed in the Korean Intellectual Property Office on November 22, 2023, the disclosure of which is incorporated herein by reference for all purposes. Technical Field

[0003] The following description relates to a method and apparatus for a traffic light recognition model. Background Art

[0004] A system in which a car, airplane, robot, or other platform operates autonomously without real-time instructions from humans can be referred to as an autonomous driving system. The implementation and operation of an autonomous driving system can generally be divided into three stages: identification, determination, and control. The identification stage is used to identify the surrounding environment using sensors mounted to the platform, the determination stage is used to determine the behavior of the platform and generate a driving path by using the identification results, and the control stage can be used to control the platform so that the platform moves along the generated path. In the identification stage, the traffic light recognition results of the autonomous vehicle can be used to set the path of the autonomous vehicle. Since there are many traffic lights in urban driving situations, accurate identification of traffic lights can be beneficial to efficient driving. Summary of the invention

[0005] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0006] In one general aspect, a method of operating an object recognition model configured to recognize a traffic light object includes: obtaining an input image from a camera included in a vehicle, the input image being among frames captured by the camera including a previous frame; estimating a first region of interest (RoI) for the input image based on prior information about the traffic light object; determining a second RoI based on the first RoI and based on a detection result of the previous frame, wherein the detection result corresponds to a recognition result of recognizing a traffic light in the previous frame by the object recognition model; and recognizing, by the object recognition model, the traffic light in the input image, wherein the recognition is based on the input image and the second RoI.

[0007] Recognizing the traffic light may include: obtaining a target image by extracting a portion of the input image corresponding to the second RoI; and enlarging the target image to a size corresponding to a size of the input image, wherein recognition by the object recognition model is also based on the enlarged target image.

[0008] The prior information about the traffic light object may include distribution data corresponding to a distribution of previously observed positions of traffic lights selected based on having a size less than a threshold.

[0009] The prior information about the traffic light object may further include: information related to the first RoI generated based on the distribution data.

[0010] The prior information about the traffic light object may include driving environment information related to the environment in which the vehicle is driving and specifications of a camera.

[0011] Estimating the first RoI may also include adjusting the first RoI based on information from sensors of the vehicle.

[0012] Estimating the first RoI may further include: adjusting the first RoI based on specifications of the camera.

[0013] Determining the second RoI may include: aggregating detection results of previous frames; obtaining a center of a cluster of detection results of previous frames; filtering the center; and obtaining the second RoI based on the filtered center.

[0014] Aggregating the detection results of the previous frames may include maintaining the detection results of a rolling window of the previous frames, the rolling window comprising a time window or a window defined by a plurality of previous frames.

[0015] Determining the second RoI may include: obtaining a distance function between centers; removing centers that do not satisfy a minimum condition in the distance function; and determining the second RoI based on centers that satisfy the minimum condition.

[0016] In another general aspect, an electronic device may include: a camera; one or more processors; and a memory storing instructions configured to cause the one or more processors to perform the following operations: obtain an input image from the camera; estimate a first region of interest (RoI) for the input image based on prior information about a traffic light object; determine a second RoI based on the first RoI and based on a detection result of a previous frame of the camera, wherein the detection result of the previous frame corresponds to a recognition result of a traffic light in the previous frame by an object recognition model, the object recognition model being configured to recognize the traffic light; and recognize the traffic light in the input image by the object recognition model, wherein the recognition is based on the input image and the second RoI.

[0017] The instructions may also be configured to cause the one or more processors to perform the following operations: obtaining a target image by extracting a portion of the input image corresponding to the second RoI, and enlarging the target image to a size corresponding to a size of the input image, and wherein the recognition performed by the object recognition model is also based on the enlarged target image.

[0018] The prior information may include distribution data corresponding to a distribution of previously observed locations of traffic lights selected based on having a size less than a threshold; and wherein the prior information may further include: information related to the first RoI generated based on the distribution data.

[0019] The electronic device may be included in a vehicle, and the instructions may be further configured to cause the one or more processors to perform the following operations: based on sensors of the vehicle and according to a driving environment of the vehicle, adjusting the first RoI.

[0020] The instructions may also be configured to cause the one or more processors to perform the following operations: aggregate the detection results of the previous frames in their time order, obtain centers of clusters of the detection results of the previous frames, and remove some of the centers, and determine the second RoI based thereon.

[0021] The instructions may also be configured to cause the one or more processors to perform the following operations: updating the detection results of the previous frame to include the results of identifying the traffic light in the input image, and the updated detection results may be used to identify the traffic light in the next input image captured by the camera.

[0022] The instructions may also be configured to cause the one or more processors to obtain a distance function between centers, remove centers other than centers satisfying a minimum condition in the distance function, and place a second RoI with respect to the remaining centers satisfying the minimum condition.

[0023] In another general aspect, a method performed by a computing device includes receiving input images captured by a camera of a moving vehicle; determining a first region of interest (RoI) for a first one of the input images, wherein the first RoI is determined based on historical traffic light observations; inputting the first input image and a zoomed-in image of an area in the first input image defined by the first RoI to an object recognition model configured to identify traffic lights, the object recognition model inferring a location of the first traffic light in the first input image based on the first input image and the zoomed-in image; determining a second RoI for a second one of the input images captured after the first input image, wherein the second RoI is determined based on the location of the first traffic light in the first input image; and inputting the second input image and the zoomed-in image of the area in the second input image defined by the second RoI to the object recognition model, the object recognition model inferring a location of the first traffic light or a second traffic light in the second input image based on the second input image and the zoomed-in image.

[0024] The method may further include adjusting a position of the second RoI based on the first RoI or based on historical traffic light observations.

[0025] The object recognition model may include a neural network, and the inferred location may include a corresponding bounding box.

[0026] Other features and aspects will become apparent from the following detailed description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1A and Figure 1B Each shows an example of operating an object recognition model according to one or more embodiments.

[0028] Figure 2 Example methods of object recognition models in accordance with one or more embodiments are shown.

[0029] Figure 3 An example of traffic light detection results is shown in accordance with one or more embodiments.

[0030] Figure 4 An example distribution of target traffic lights is shown in accordance with one or more embodiments.

[0031] Figure 5 An example of a region of interest (RoI) update relative to an input image is shown in accordance with one or more embodiments.

[0032] Figure 6 Example operations of a detector according to one or more embodiments are shown.

[0033] Figure 7 An example of an input image in which a RoI is changed is shown in accordance with one or more embodiments.

[0034] Fig. 8A and Figure 8B 2 and 3 show examples of traffic light detection results according to one or more embodiments.

[0035] Fig. 9 An example of a comparison of traffic light detection results is shown in accordance with one or more embodiments.

[0036] Fig.10 An example electronic device is shown in accordance with one or more embodiments.

[0037] Throughout the drawings and detailed description, unless otherwise described or provided, the same or similar reference numerals will be understood to refer to the same or similar elements, features, and structures. The drawings may not be drawn to scale, and the relative sizes, proportions, and depictions of elements in the drawings may be exaggerated for clarity, illustration, and convenience. DETAILED DESCRIPTION

[0038] The following detailed description is provided to help the reader obtain a comprehensive understanding of the methods, devices and / or systems described herein. However, after understanding the disclosure of the present application, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be apparent. For example, the order of operations described herein is merely an example and is not limited to those order of operations set forth herein, but may be significantly changed after understanding the disclosure of the present application, except for operations that must be performed in a certain order. Similarly, for greater clarity and brevity, descriptions of features known after understanding the disclosure of the present application may be omitted.

[0039] The features described herein may be implemented in different forms and should not be construed as being limited to the examples described herein. Rather, the examples described herein are provided merely to illustrate some of the many possible ways to implement the methods, devices and / or systems described herein, which will be apparent after understanding the disclosure of the present application.

[0040] The terms used in this article are only used to describe various examples and are not used to limit the present disclosure. Unless the context clearly indicates otherwise, the articles "one", "an" and "the" are also intended to include plural forms. As used herein, the term "and / or" includes any one of the items listed in the association and any combination of any two or more. As non-limiting examples, the terms "comprise", "include" and "have" represent the presence of the features, quantities, operations, components, elements, and / or combinations thereof, but do not exclude the presence or addition of one or more other features, quantities, operations, components, elements, and / or combinations thereof.

[0041] Throughout the specification, when a component or element is described as being "connected to," "coupled to," or "engaged to" another component or element, it may be directly "connected to," "coupled to," or "engaged to" the other component or element, or one or more other components or elements may reasonably be present in between. When a component or element is described as being "directly connected to," "directly coupled to," or "directly engaged to" another component or element, there may not be other elements in between. Similarly, for example, "between" and "directly between," as well as "adjacent to" and "immediately adjacent to" may also be interpreted as described above.

[0042] Although terms such as "first", "second" and "third", or A, B, (a), (b) may be used herein to describe various components, assemblies, regions, layers or parts, these components, assemblies, regions, layers or parts are not limited by these terms. For example, each of these terms is not used to define the nature, order or sequence of the corresponding component, component, region, layer or part, but is only used to distinguish the corresponding component, component, region, layer or part from other components, components, regions, layers or parts. Therefore, without departing from the teachings of the examples described herein, the first component, component, region, layer or part mentioned in the examples may also be referred to as the second component, component, region, layer or part.

[0043] Unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as those commonly understood by those of ordinary skill in the art to which the present disclosure belongs, and as understood based on the understanding of the disclosure of the present application. For example, terms defined in commonly used dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the disclosure of the relevant technology and the present application, and should not be interpreted as an ideal or overly formal meaning, unless explicitly defined as such herein. In this article, the use of the term "may" with respect to examples or embodiments (for example, with respect to what an example or embodiment may include or implement) indicates that there is at least one example or embodiment that includes or implements such features, and all examples are not limited thereto.

[0044] Object detection models and algorithms are generally unreliable when detecting tiny objects in images. For example, in a driving system intended for use in a vehicle driving on a road, the detection of tiny objects is generally the same as the detection of distant objects (due to the perspective effect) for a moving vehicle, which is the same as early object detection. In a driving system, it may be beneficial to accurately detect distant objects as quickly as possible because early detection of smaller objects in the distance can enable more accurate and efficient decision-making, path planning, etc. However, in terms of perspective, the farther the object is from the distance, the smaller the object is in the image captured by the camera. Traffic lights are a type of object that is usually used as a detection target in a driving system. The detection of traffic lights is very important for driving functions, and early detection may be beneficial for this. In a typical vehicle sensing system (camera, radar, etc.), a front camera can be used to capture images in the vehicle's driving path. In this example scenario, the detection of traffic lights about 70 to 100 meters in front of the vehicle can be particularly beneficial. However, for a typical camera, at this distance, the image of the traffic light in the driving image may be tiny in the driving image and is therefore difficult to detect.

[0045] For example, a tiny object may be, for example, an object of about 16 pixels (px) x 16 pixels or less (although in practice, the size at which detection becomes unreliable will vary from system to system). Since the number of pixels representing the object is small, the image information is small, and therefore, the object may not be suitable for object detection because the object is unlikely to be distinguished from the background. For example, one of the reasons may be related to how object detection is typically performed. An object detection model / algorithm may determine a bounding box of an object in an image. Since there are typically many bounding boxes, techniques have been used to merge or filter redundant or unwanted bounding boxes even for the same object in an image. For example, intersection over union (IoU) is a known measure of the degree of overlap of two bounding boxes. Training of an object detection model may involve calculating the IoU of a true (GT) bounding box with a predicted bounding box. However, for smaller objects, a small deviation (e.g., 2 pixels) between the GT box and the predicted box can significantly reduce the IoU. Due to the rigidity of IoU and similar methods, object detection (and training for object detection) of distant / smaller objects may be poor, and smaller objects are often not detected or not well trained. That is, even a slight difference in predictions may reduce the Intersection over Union (IoU) and may not be easy to train / detect. In addition, even when humans manually label smaller objects for training, these objects may be easily mislabeled or missed, which may also reduce the training effect.

[0046] In order to solve the problem of smaller object detection, learning-based methods can be used to improve or emphasize the training and performance of object recognition models. However, learning-based methods suitable for focusing on detecting tiny objects may become worse when detecting larger objects, or have other disadvantages. Since larger objects such as vehicles may need to be detected together with smaller objects such as traffic lights in driving situations, the overall performance of the detection model focused on smaller objects may decline and may cause defects in autonomous vehicles. In addition, in order to ensure that the learning-based method operates properly, it may be necessary to accurately give the true labels for tiny objects. However, as mentioned above, in reality, tiny objects may be mislabeled or omitted. In addition, there may be no training data to illustrate new driving situations. Accordingly, although autonomous vehicles need to adapt to the current situation, previous learning-based methods are generally limited in data robustness.

[0047] The description of the embodiments and examples disclosed herein may include methods for detecting smaller objects (e.g., traffic lights) by an object recognition model used in a driving situation. Although for the sake of description, traffic lights are referred to herein as target objects (targets for object detection), the embodiments and examples described herein may be applied to any type of smaller object, particularly when such objects have predictable / consistent statistical appearance patterns in driving images, and / or when the target object may move across a series of frames (e.g., due to vehicle movement). In addition, although autonomous driving is mentioned as an example application, the embodiments and techniques described herein may be used in any object detection scenario or application where smaller object detection may be expected.

[0048] Figure 1A and Figure 1B Each shows an example of operating the object recognition model 100 according to one or more embodiments.

[0049] like Figure 1A and Figure 1B As shown in, one or more blocks and combinations of these blocks can be implemented by a computer based on dedicated hardware that performs a predetermined function, or a combination of computer instructions and dedicated hardware. For example, hardware can be configured for any purpose and function by loading and executing processor executable instructions.

[0050] refer to Figure 1A and Figure 1B A "block" or "module" may be a unit that includes, for example, one of hardware, software, or firmware, or a combination of two or more. For example, the term "block" may be used interchangeably with terms such as unit, logic, logic block, component, or circuit. A block may be the smallest unit of an integrally formed component or a part thereof. A block may be the smallest unit for performing one or more functions or a part thereof. A block may be a unit of execution abstraction or data abstraction (e.g., a neural network model or its components, an image, a feature map, an inference result, etc.). For example, a block may include any one or any combination of an application specific integrated circuit (ASIC) chip, a field programmable gate array (FPGA), a programmable logic device, or an algorithm driven by a processor 1030 that performs known operations or operations to be developed.

[0051] For example, systems for autonomous vehicles (e.g. Fig.10 The electronic device 1000 may include a processor 1030 (eg, Fig.10 The processor 1030 may perform an operation 100-1 of estimating a region of interest (RoI) and an operation 100-2 of estimating an object according to a result of the operation 100-1.

[0052] The object recognition model can be used to detect and identify (recognize) specific / target objects in an image or video. The object recognition model can accurately determine the location and type of the object by combining deep learning with computer vision techniques. The object recognition model can use any of various types of known object recognition models as constituent components (e.g., "Look Only Once" (YOLO), Faster Region-based Convolutional Neural Network (R-CNN), or Single Shot Multibox Detector (SSD) to name some examples (e.g., for detector 140). The processor 1030 of the autonomous vehicle can drive the object recognition model so that it can be used efficiently in real time.

[0053] In the following description, it may be assumed that the object recognition model includes a traffic light detection model that has been (or is being) trained by traffic light training data.

[0054] The processor 1030 may input the input image 101, the prior knowledge 102, and the previous frame detection result 161 to the RoI estimator 110, and based thereon (e.g., one or the other), the RoI 111 may be obtained. The processor 1030 may input the input image 101 and the RoI 111 to the target image generator 120, and the target image generator 120 may generate a target image 113 based on the corresponding RoI 111 (one target RoI and one target image are possible). Figure 1B The RoI 111 in is shown relative to the input image 101 ; in practice, the RoI estimator may not output an image, but rather the RoI in the form of, for example, its size and position (relative to the input image 101 ).

[0055] The target image 113 may be, for example, an enlarged image of the RoI 111 obtained (cropped) from the input image 101 (as input to the target image generator 120). That is, a sub-image of the input image 101 defined by (within) the corresponding RoI 111 may be extracted and enlarged to form the target image 113 (capable of matching the size of the input image 101). The processor 1030 may aggregate the input image 101 and the target image 113 into an aggregated image 130 (e.g., by cascading). The aggregated image 130 may be input to the detector 140. The processor 1030 may apply the detector 140 to the aggregated image 130 to generate a detection result 141 output from the detector 140.

[0056] The detection results 141 may be input to a detection result aggregator 150, which may generate and output a final detection result 151 based thereon, and as briefly described below, information of the final detection result 151 may assist in RoI estimation performed by RoI estimator 110 on input images subsequent to input image 101. The final detection result 151 may be used for various driving functions described above, such as path planning, decision making, assisted driving, displaying information to the driver, and the like. Additionally, processor 1030 may add at least some of the final detection results 151 to a set of previous frame detection results 161, which may be accomplished by inputting the final detection results 151 to a detection result accumulator 160, which may decide to merge a portion of the final detection result 151 (if any) into the previous frame detection result 161, and may also "roll down" the previous frame detection result as it expires (becomes too old). The previous frame detection result 161 may include final detection results corresponding to multiple frames prior to the input image 101. For example, the previous frame detection results 161 may correspond to a moving window of previous detection results (e.g., the 10 most recent frames). However, the previous frame detection results 161 may become empty due to a lack of detection results (e.g., no traffic light was detected), or the detection result accumulator 160 repeatedly does not select any of the final detection results to be included in the previous frame detection results 161. In this case, the RoI estimator 110 may estimate the RoI without any previous frame detection results (e.g., based only on the prior knowledge 102). When the previous frame detection results are available, they may be used to adjust or update the baseline RoI of the prior knowledge 102. Reference Figure 2 The operation of the above-mentioned processor 1030 is described in detail.

[0057] Figure 2 Example methods of object recognition models in accordance with one or more embodiments are shown.

[0058] refer to Figure 1A and Figure 1B The descriptions provided are generally applicable to Figure 2 .

[0059] For ease of description, the following describes Fig.10 The processor 1030 shown in FIG. 1 performs operations 210 to 240. However, operations 210 to 240 may be performed by another suitable electronic device in any suitable system.

[0060] As an overview, Figure 2Operations 220 and 230 are shown for using prior knowledge and results from previous frames, respectively, to form a RoI for focusing on object detection in operation 240. In general, for a given input image, operation 220 may be performed to generate a first RoI based on prior knowledge (a baseline / default assumption about statistically likely image locations of smaller / distant objects such as traffic lights), and operation 230 may be performed to summarize the locations of recent object detections and update (e.g., move, omit, merge, etc.) the first RoI to form a second RoI for object detection. If there are not enough recent object detections (e.g., previous frame detection results 161) for a given input image, the first RoI may be used without adjustment. Regardless of whether there are previous frame detection results, the first / default RoI may be updated (e.g., moved, resized, etc.) based on current driving conditions (e.g., the pose of the vehicle, the location of the vehicle, etc.).

[0061] Return to Figure 2 , in operation 210, the processor 1030 may obtain an input image 101 from a camera included in an autonomous vehicle. The camera may be / include various types of cameras. For example, the camera may continuously sense the vicinity of the vehicle and may collect important information during driving to assist vehicle driving. The autonomous vehicle may include a front camera, a rear camera, an internal camera, and / or a night vision camera (e.g., infrared). In this case, the autonomous vehicle may typically use a front camera to detect traffic lights, and the input image 101 may come from the front camera. The front camera may be a camera mounted in front of the vehicle, and may capture driving images and videos to monitor road conditions and detect, for example, obstacles, traffic lights, or lanes in front of the vehicle. To this end, the processor 1030 may obtain an input image from the front camera and may perform object recognition in real time.

[0062] In operation 220, the processor 1030 according to an embodiment may estimate a first RoI (eg, Figure 4 The RoI estimator 110 according to an embodiment may estimate the first RoI in the input image 101 using prior knowledge as input.

[0063] Prior knowledge 102 may include, be based on, or reflect statistical distribution data of previously recorded image positions of target traffic lights. Typically, prior knowledge 102 may be a statistical compilation of specific types of objects (e.g., traffic lights) most often "seen" by a specific camera pose of a vehicle camera, possibly limited to "sightings" that satisfy specific conditions related to the expected object detection scenario (e.g., detecting a traffic light ahead within a certain distance range). More specifically, prior knowledge 102 may be obtained by extracting / compiling a distribution of image positions of target traffic lights whose pixel values ​​are less than or equal to a threshold value (e.g., distribution 400 of target traffic lights) from image data obtained from driving history (e.g., a test drive environment, early / remote observations captured from a fleet of vehicles that were subsequently confirmed as real data, etc.). The inclusion threshold or condition may be that the pixel value / size of the target traffic light must be less than or equal to a value included in the prior knowledge 102 or contributing to the prior knowledge 102 (which may depend on the characteristics of the camera of the autonomous vehicle). For example, a target traffic light 70 to 100 meters (m) away from the autonomous vehicle may be identified as approximately 16px in the camera's image, and therefore, 16px may be set as the threshold value. In addition, the prior knowledge 102 may be / include information related to the first RoI generated based on the distribution data of the target traffic light. In other words, the prior knowledge 102 may be the first RoI itself, which is generally static (although it may be exchanged and / or adjusted as described elsewhere herein). Figure 4 Describe the distribution of target traffic lights in detail.

[0064] Although prior knowledge 102 may generally remain static, prior knowledge 102 can be updated in response to changing driving conditions. For example, since traffic lights may be placed (relative to the road) and mounted (e.g., high or low, overhead or corner, etc.) in different ways, there may be instances of prior knowledge 102 for corresponding geographic areas, specifically, areas with different statistical distributions of traffic lights. Information on the current geographic location of the autonomous vehicle may be used to exchange prior knowledge 102 corresponding to the current location.

[0065] Prior knowledge 102 may include driving environment information related to the environment in which the autonomous vehicle is driving, and specifications of a camera included in the autonomous vehicle, which may be used to more accurately generate or adjust the first RoI according to prior knowledge 102 .

[0066] More specifically, the processor 1030 according to an embodiment may facilitate the correction / adjustment of the first RoI (based on the prior knowledge 102) by using a sensor (e.g., a tilt / orientation sensor such as a gyroscope) that senses the driving environment of the autonomous vehicle, or the state of the autonomous vehicle (e.g., 3D posture). In addition, the processor 1030 may facilitate the correction / adjustment of the first RoI based on the specifications of the camera included in the autonomous vehicle. In one approach, such driving environment information related to the vehicle surroundings / environment, camera, vehicle state, etc. may be used to adjust the prior knowledge 102 (e.g., in the form of RoI) so that the prior knowledge 102 better corresponds to the input image 101 (i.e., so that the statistical information represented in the prior knowledge 102 is consistent with the input image 101). For example, if the sensor indicates that the autonomous vehicle is tilted forward (downhill), the first RoI may be adjusted / generated accordingly (e.g., moved upward) to place the first RoI (relative to the input image) at a location where a traffic light is statistically expected to appear at that viewing / camera angle. In summary, dynamic and static driving environment information may be provided to the RoI estimator 110 as part of (or in addition to) the prior knowledge 102 , which may adjust the estimated RoI accordingly.

[0067] In detail, the driving environment information may include road and lane information, road status and weather information, a road geographic information system (GIS), navigation data, or information related to road traffic rules or the installation position of a traffic light. As described above, the RoI estimator 110 may estimate the first RoI by using the driving environment information input as a supplement to the prior knowledge 102. For example, when information that the installation position of a traffic light on the current driving road is higher than a general position is input, the first RoI input as the prior knowledge 102 may be corrected so that the first RoI may be estimated to be a little higher.

[0068] The specifications of a camera for an autonomous vehicle may vary by manufacturer or model, but may be predetermined prior to installation in the autonomous vehicle. The specifications of a camera may typically include resolution, frame rate, lens type and field of view, low light performance, auto focus and adjustment, or data transfer rate per second.

[0069] although Figure 1B The point images are shown as input to the RoI estimator 110 as prior knowledge 102, but in practice, the object recognition model 100 can pre-calculate the RoIs based on statistical prior knowledge (historical traffic positions) and reuse those pre-calculated RoIs as the first RoIs. The RoI estimator 110 can update / adjust those first RoIs based on the current environment and / or state of the autonomous vehicle. Such adjustments can include moving the first RoIs to align them with the current input image.

[0070] In addition, the RoI estimator 110 may correct the first RoI according to the specifications of the camera. The prior knowledge 102 may include distribution data of target traffic lights predetermined according to the specifications of the camera and information related to the first RoI. However, for example, when the processor 1030 determines that the camera has a problem in a real-time driving environment, or the problem is caused by a day / night change in resolution, the RoI estimator 110 may estimate the first RoI by adjusting the position and size of the first RoI as input of the prior knowledge 102.

[0071] In operation 230, based on the previous frame detection result 161 (when available), the processor 1030 may update the first RoI to a second RoI (eg, Figure 5 For example, as described below, the center point of a cluster of previous object detections (e.g., previous frame detection results 161) can be used to guide the first RoI towards the area of ​​the most recent object detection by the object detection model.

[0072] Using the first RoI estimated based on the prior knowledge 102, the success rate of object detection for detecting the target traffic light may be sufficient. However, the first RoI is based on the complete statistical data distribution of the driving history (e.g., the test drive image), and therefore may cover the area or part of the current input image that does not currently require object detection. Unnecessary areas / parts may add unnecessary calculations, and therefore, the operation speed of the object recognition model 100 may be reduced. Generally, the frames of the driving image are changed sequentially, and therefore, the position, size, and number of the first RoI (to form the second RoI) may be adjusted in real time by reflecting the detection results of the previous frame. In other words, the RoI estimator 110 can adjust the first RoI, and the first RoI can be updated to the second RoI.

[0073] The processor 1030 according to an embodiment may obtain the previous frame detection results 161 detected by the detector 140 in order from the latest. The processor 1030 may accumulate and store the previous frame detection results 161 until a predetermined number of final detection results 151 (of the corresponding previous input image) are accumulated, and may then begin to discard sufficiently old previous frame detection results.

[0074] The detection result accumulator 160 may accumulate and aggregate the final detection result 151 in order from the most recent. In this case, the detection result accumulator 160 may store the final detection result 151 until the number of previous frame detection results 161 to be used in the RoI estimator 110 is greater than or equal to a predetermined number N (e.g., 50 frames). When the number of previous frame detection results 161 of the detection result accumulator 160 is less than or equal to a predetermined number (e.g., N or a number less than N), the RoI estimator 110 may estimate the RoI by using only the prior knowledge 102. That is, the RoI estimator 110 may utilize previous frame detection results less than a maximum number, and may use previous frame detection results only when there is a same threshold minimum number.

[0075] The processor 1030 according to an embodiment may apply a clustering algorithm to the detection results of the previous frame to obtain a representative center of the cluster of the previous detection results. When the number of the previous frame detection results 161 is greater than or equal to the predetermined number N, the RoI estimator 110 may update the first RoI to the second RoI. The operation of updating to the second RoI may be performed by the method described below.

[0076] The RoI estimator 110 may obtain K representative centers by a clustering algorithm for the center points of the N detection results. In other words, the detection result 141 may be represented in the form of a bounding box, and therefore, the clustering algorithm may be applied to the center point of the bounding box of the detection result 141. To name some non-limiting examples, the clustering algorithm may be a K-means algorithm or a density-based spatial clustering of noise applications (DBSCAN) algorithm.

[0077] The processor 1030 may filter the representative centers (eg, using IoU) and obtain a second RoI.

[0078] The processor 1030 according to an embodiment may obtain a distance function between representative centers. The processor 1030 may remove representative centers that do not satisfy a minimum condition in the distance function. The processor 1030 may generate a second RoI around the representative center that satisfies the minimum condition.

[0079] The RoI estimator 110 may perform filtering on the centers to reduce areas where RoIs for multiple representative centers excessively overlap with each other ( Figure 5 First, a distance function for each representative center can be obtained to perform filtering. Each center can be represented by C = {c 1 , c 2 , ..., c N} indicates that, and d(c i , c k) can be defined as a distance function between two centers (e.g., Euclidean distance). Euclidean distance can be a straight-line distance between two points. Euclidean distance can be a generalization of the Pythagorean theorem. Euclidean distance can be used to measure the "shortest distance" between two points. For example, Euclidean distance can also be used to measure the similarity between clusters in the K-means clustering algorithm.

[0080] When the center c among the representative centers i , When c i is designated as a candidate for removal. In this case, the center that satisfies the minimum condition can be assumed to be c j If the minimum condition is not met, c can be retained i .

[0081] When targeting c i , When , the RoI estimator 110 may remove the c designated as a candidate for removal. j , and c can be removed in the opposite case i The RoI estimator 110 may repeat the above process while increasing from i=1 to i=N.

[0082] The RoI estimator 110 may generate a second RoI based on the representative center filtered by repeating the above process. The RoI estimator 110 may update the first RoI according to the generated second RoI, and may output an input image 111 designated by the RoI.

[0083] The centers of the filtered previous detection results can be used in a variety of ways to form the final (e.g., second) RoI of the target image generator 120. In one embodiment, when sufficient previous frame results are available, the filtered center can be used alone as a basis for determining the location of the second / final RoI. For example, the second RoI can be centered on the filtered center and can have a predetermined size (or a size based on prior knowledge). In another embodiment, the second RoI can be a weighted mixture of the first RoI (based on prior knowledge) and the filtered center, where the filtered center gets more weight as the number of previous frame results increases. In another embodiment, the previous frame results can be in the form of a detection result bounding box (e.g., of an identified traffic light), which can be enlarged and mixed with the first RoI. In yet another embodiment, when there are no / insufficient previous frame results, the first RoI is used as the second / final RoI, and when the filtered center is available, the RoI based on the filtered center can be used as the second RoI.

[0084] In operation 240 , the processor 1030 may detect a traffic light in the input image 101 based on the input image 101 and the second RoI by using the detector 140 .

[0085] The processor 1030 according to an embodiment may obtain a target image initially having a size corresponding to the size of the second RoI. The processor 1030 may enlarge the target image to the size of the input image. The processor 1030 inputs the input image 101 and the enlarged target image to the detector 140 for object detection.

[0086] The target image generator 120 may crop (copy) a region (sub-image) in the input image 101 where the second RoI is the target for each second RoI. The target image generator 120 may enlarge (e.g., upsample) the cropped image (sub-image) to the size of the input image, and may output a target image (e.g., Figure 6 The target image 132, or Figure 1B target image 113).

[0087] Image aggregator 130 (see Figure 6 ) The input image 101 and the target image may be aggregated, and the aggregated image may be input to the detector 140. The target image may be formed for each RoI separately. For example, the aggregation may be performed by concatenating the input image and the target image.

[0088] The detector 140 may receive the input image 101 and the target image from the image aggregator 130 (in aggregated form), and may output a detection result. The detector 140 may be / include a model configured to detect or recognize traffic lights from the input image (e.g., by previous training). Accordingly, the detector 140 may detect traffic lights from the input image 101 and the target image. The detector 140 may perform detection of traffic lights on the input image 101 and the target image in parallel or in the order of input, and may also perform object detection for detecting vehicles, lanes, etc. together. The detection result 141 output from the detector 140 may be sent to the detection result aggregator 150. The detection result 141 may be, for example, a bounding box with a corresponding classification label for the detected object.

[0089] The detection result aggregator 150 may aggregate the detection results 141 of the detector 140 and may output a final detection result 151. Since the target image generated by enlarging the second RoI is substantially a portion of the corresponding input image, there may be overlapping detection results between the detection in the input image and the detection in the target image. Accordingly, the detection result aggregator 150 may execute an algorithm (e.g., a non-maximum suppression algorithm) for removing such overlapping results. The final detection result 151 output from the detection result aggregator 150 may be relayed to the detection result accumulator 160 and may be used to generate the next second RoI 110 through the RoI estimator 110 and included in the previous frame detection result 161.

[0090] Figure 3 An example of traffic light detection results is shown in accordance with one or more embodiments.

[0091] refer to Figures 1A to 2 The descriptions provided are generally applicable to Figure 3 .

[0092] refer to Figure 3 , an object recognition model of general performance may have difficulty detecting a traffic light 310 that is far away. In other words, a traffic light 310 that is about 70 to 100m away from the vehicle may be captured as an object of the traffic light 310 of about 10px in an ordinary camera. A method of performing object detection to find tiny objects may involve image magnification. However, detecting the traffic light 310 by magnifying the entire image instead of a limited area may be inefficient for vehicle functions that need to respond to changing driving conditions in real time (e.g., autonomous driving). Accordingly, only by performing the above-referenced method, the traffic light 310 may be detected. Figures 1A to 2 The described enlarged RoI to detect the traffic light 310 may be efficient.

[0093] Figure 4 An example distribution of target traffic lights is shown in accordance with one or more embodiments.

[0094] refer to Figures 1A to 2 The descriptions provided are generally applicable to Figure 4 .

[0095] The prior knowledge 102 may include data related to the distribution 400 of target traffic lights, which is obtained by extracting the distribution 400 of target traffic lights whose pixel values ​​(e.g., areas) are less than or equal to a threshold value from image data obtained in a test drive environment. In addition, the prior knowledge 102 may include information related to the first RoI (e.g., regions 411, 412, and 413) generated based on the data related to the distribution 400 of target traffic lights.

[0096] Prior knowledge 102 may be a scenario / task specific knowledge. For an autonomous vehicle to quickly respond to traffic flow in real time under driving conditions, detecting distant traffic lights is helpful. Accordingly, areas 411, 412, and 413 where distant target traffic lights are mainly distributed may be designated as baseline (or first) RoIs.

[0097] Since traffic lights are usually installed with specific standards and rules according to traffic regulations, in a driving situation, most target traffic lights may appear in specific areas 411, 412, and 413. The areas where the target traffic lights are distributed may be determined based on external factors (installation position, viewing angle, etc.) and internal factors (focal length, etc.) of the camera.

[0098] refer to Figure 4 , the distribution 400 of target traffic lights with a height less than or equal to 15 px may be an image with a size of 2560 px×1280 px obtained in a test drive environment.

[0099] If the processor 1030 is to detect the target traffic light based on the entire image, a local area with a width and height of 1 / 4 size is actually processable. However, if only the area including the target traffic light is considered, 3 to 4 RoIs are sufficient. Accordingly, the area including the target traffic light can be defined and used as the RoI. In summary, the prior knowledge 102 may include information related to the first RoI, and the first RoI includes information related to the RoI.

[0100] Figure 5 An example of updating a RoI relative to an input image in accordance with one or more embodiments is shown.

[0101] refer to Figures 1A to 2 and Figure 4 The descriptions provided are generally applicable to Figure 5 .

[0102] The RoI estimator 110 according to an embodiment may adjust the overlapping regions 511 and 512 in the input image 510 to which the first RoI is specified, and may output the input image 520 to which the second RoI is specified by updating the first RoI.

[0103] The RoI estimator 110 may receive the first RoIs 411, 412, and 413, and may output the input image 510 to which the first RoIs 411, 412, and 413 are specified. However, the input image 510 to which the first RoIs 411, 412, and 413 are specified is determined according to the distribution of the predetermined target traffic lights, and the first RoI 413 and the overlapping areas 511 and 512, which are areas not including the traffic lights, may occur in actual driving situations. Accordingly, the RoI estimator 110 may be configured as described above with reference to FIG. Figures 1A to 2The described method updates the first RoIs 411 , 412 , and 413 to the second RoIs 521 and 522 .

[0104] Figure 6 An example operation of the detector 140 is shown in accordance with one or more embodiments.

[0105] refer to Figures 1A to 2 The descriptions provided are generally applicable to Figure 6 .

[0106] refer to Figure 6 , the detector 140 may receive an input image 131 and a target image 132 from the image aggregator 130, and may output a detection result 141. Here, the detection result is shown with respect to the input image 131, however, in reality, the detection result 141 may be an indication of the detection of a target object (e.g., a traffic light). For example, the detection result 141 may be in the form of a bounding box and a corresponding label inferred / predicted by the detector 140. The detector 140 may include a general object recognition model, and may include a traffic light detection model for an autonomous vehicle, for example, to detect a traffic light and identify a traffic signal state (e.g., a red light, a green light, a left turn light, etc.). For example, in general, the detector 140 may receive input data, and may convert the input data (e.g., the input image 131 and the target image 132) into a format that can be input to the object recognition model of the detector 140 through preprocessing. The preprocessing may include image resizing, normalization, and / or data augmentation. The detector 140 may then extract features from the input data, and may detect and identify general objects (e.g., vehicles or lanes) based on the trained model. In this case, the detector 140 may detect and identify the object, and may provide a bounding box and a label for the detected object, as shown in the detection result 141. The detector 140 may add the bounding box and the label to the traffic light in the input image 131, as shown in the detection result 141, and may also express the result of the target image 132 using the bounding box and the label in the detection result 141.

[0107] Figure 7 An example input image is shown in which the RoI is changed in accordance with one or more embodiments.

[0108] refer to Figures 1A to 2 and Figure 4 The descriptions provided are generally applicable to Figure 7 .

[0109] refer to Figure 7, the autonomous vehicle may be driven on a flat surface 710 or an inclined surface 711. The prior knowledge 102 may not easily include distribution data of all target traffic lights observed by a vehicle driving on an inclined surface 720. That is, the statistical expected RoI of the traffic lights may not be aligned with the input image. Accordingly, based on the first RoIs 411, 412, and 413 of the vehicle driving on the flat surface 710, the processor 1030 may calibrate / adjust the first RoIs 411, 412, and 413 by using data from a sensor (e.g., a gyroscope sensor) included in the autonomous vehicle. For example, when the autonomous vehicle is driving downward (such as on an inclined surface 720), the target traffic light may be observed at an image position higher than the image position of the target traffic light observed from the flat surface 710. Accordingly, the RoI estimator 110 may calibrate / move the first RoIs 411, 412, and 413 upward (relative to the input image) based on the gyroscope sensor data received when on the inclined surface 720. Any alignment method may be used. For example, visual cues, path planning, environmental models, etc.

[0110] Fig. 8A and Figure 8B 2 and 3 show examples of traffic light detection results according to one or more embodiments.

[0111] refer to Figures 1A to 7 The descriptions provided are generally applicable to Fig. 8A and Figure 8B .

[0112] refer to Fig. 8A and Figure 8B , when the processor 1030 detects a traffic light by inputting only an input image to the detector 140, detection results 811 and 831 in which the target traffic light is not detected may be output. In this case, the autonomous vehicle may not be able to easily respond to the changing traffic environment in real time. However, when the processor 1030 generates a target image by using the input image specified by the RoI using the RoI estimator 110, and detects a traffic light by inputting the target image together with the input image to the detector 140, detection results 812 and 832 in which the target traffic light is detected may be output.

[0113] Fig. 9 An example comparison of traffic light detection results is shown in accordance with one or more embodiments.

[0114] refer to Figures 1A to 8B The descriptions provided are generally applicable to Fig. 9 .

[0115] refer to Fig. 9 ,like Fig. 8A and Figure 8BAs shown in , when the original image 900 is input to the detector 140 before the RoI estimator 110 is applied in operation 910, the processor 1030 may detect the target traffic lights when some target traffic lights are missing, or may not detect the target traffic lights at all. As shown in the original image 900, because it is not ensured that there are enough pixels in the camera, the target traffic lights that can be seen by the human eye may become too small to be verified, or may not be distinguished from the background. However, when the target image and the original image 900 are input to the detector 140 after the RoI estimator 110, the target image generator 120, and the image aggregator 130 are applied in operation 920, all target traffic lights can be detected and even their labels can be output.

[0116] Fig.10 An example electronic device is shown in accordance with one or more embodiments.

[0117] refer to Fig.10 , the electronic device 1000 may include a processor 1030, a memory 1050, and an output device 1070 (e.g., a display). The processor 1030, the memory 1050, and the output device 1070 may be connected to each other via a communication bus 1005. The processor 1030 may be any one or any combination of the different types of processors mentioned below.

[0118] The output device 1070 may display the input image in which the RoI is designated by the processor 1030 and the traffic light detection result together with the user interface.

[0119] The memory 1050 may store an object recognition model executed in the processor 1030. In addition, the memory 1050 may store various information generated in the operation of the above-mentioned processor 1030. In addition, the memory 1050 may store various data, programs, etc. The memory 1050 may include a volatile memory or a non-volatile memory. The memory 1050 may include a large-capacity storage medium such as a hard disk, and store various data.

[0120] In addition, the processor 1030 may execute reference Figures 1A to 9 The at least one method described, or an algorithm corresponding to the at least one method. The processor 1030 may be a data processing device implemented by hardware including a circuit having a physical structure for performing the desired operation. For example, the desired operation may include a code or instruction in a program. The processor 1030 may be implemented as, for example, a central processing unit (CPU), a graphics processing unit (GPU), or a neural network processing unit (NPU). The hardware-implemented electronic device 1000 may include, for example, a microprocessor, a CPU, a processor core, a multi-core processor, a multiprocessor, an ASIC, and an FPGA.

[0121] The processor 1030 may execute a program and control the electronic device 1000. Codes of the program executed by the processor 1030 may be stored in the memory 1050.

[0122] Computing device, vehicle, electronic device, processor, memory, image sensor, vehicle / operating function hardware, ADAS / AD system, display, information output system and hardware, storage device, and the like Figures 1A to 10Other devices, equipment, units, modules and components described are implemented or represented by hardware components. Examples of hardware components that can be used to perform the operations described in the present application include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in the present application, where appropriate. In other examples, one or more hardware components performing the operations described in the present application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer can be implemented by one or more processing elements (e.g., logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field programmable gate arrays, programmable logic arrays, microprocessors, or any other device or combination of devices configured to respond and execute instructions in a defined manner to achieve the desired result). In one example, a processor or computer includes (or is connected to) one or more memories storing instructions or software executed by a processor or computer. The hardware components implemented by a processor or a computer can execute instructions or software, for example, an operating system (OS) and one or more software applications running on the OS to perform the operations described in this application. The hardware components can also access, manipulate, process, create and store data in response to the execution of instructions or software. For the sake of brevity, the singular term "processor" or "computer" can be used in the description of the examples described in this application, but multiple processors or computers can be used in other examples, or the processor or computer can include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components can be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components can be implemented by one or more processors, or a processor and a controller, and one or more other hardware components can be implemented by one or more other processors or another processor and another controller. One or more processors, or a processor and a controller can implement a single hardware component, or two or more hardware components. The hardware components may have any one or more of different processing configurations, examples of which include a single processor, independent processors, parallel processors, single instruction single data (SISD) multiprocessing, single instruction multiple data (SIMD) multiprocessing, multiple instruction single data (MISD) multiprocessing, and multiple instruction multiple data (MIMD) multiprocessing.

[0123] Perform the operations described in this application Figures 1A to 10The method shown in is performed by computing hardware, for example, by one or more processors or computers implemented as described above, implementing instructions or software to perform the operations described in the present application (operations performed by the method). For example, a single operation, or two or more operations can be performed by a single processor, or two or more processors, or a processor and a controller. One or more operations can be performed by one or more processors, or a processor and a controller, and one or more other operations can be performed by one or more other processors, or another processor and another controller. One or more processors, or a processor and a controller can perform a single operation, or two or more operations.

[0124] For controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the instructions or software of the method as described above can be written as a computer program, code segment, instruction or any combination thereof, for indicating or configuring one or more processors or computers individually or collectively to operate as a machine or a special-purpose computer to perform the operations performed by the above-mentioned hardware components and methods. In one example, the instructions or software include machine codes directly executed by one or more processors or computers, such as machine codes generated by a compiler. In another example, the instructions or software include more advanced codes executed by one or more processors or computers using an interpreter. Instructions or software can be written in any programming language based on the block diagrams and flow charts shown in the accompanying drawings, and the corresponding descriptions herein (which disclose algorithms for performing the operations performed by the hardware components and the methods as described above).

[0125] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above, as well as any associated data, data files, and data structures, may be recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), flash memory, card-type storage such as multimedia card micro or card (e.g., secure digital (SD) or extreme digital (XD)), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk, and any other device configured as follows: store instructions or software and any associated data, data files and data structures in a non-temporary manner, and provide instructions or software and any associated data, data files and data structures to one or more processors or computers so that one or more processors or computers can execute the instructions. In one example, the instructions or software and any associated data, data files and data structures are distributed on a networked computer system so that one or more processors or computers store, access and execute the instructions and software and any associated data, data files and data structures in a distributed manner.

[0126] Although the present disclosure includes specific examples, it will be apparent after understanding the disclosure of the present application that various changes in form and detail may be made to these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein should be considered to be descriptive only and not for limiting purposes. The description of features or aspects in each example is considered to be applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order and / or if components in the described systems, architectures, devices, or circuits are combined in different ways and / or replaced or supplemented by other components or their equivalents.

[0127] Therefore, the scope of the disclosure may be defined by the claims and their equivalents in addition to the above disclosure, and all variations within the scope of the claims and their equivalents are to be construed as being included in the disclosure.

Claims

1. A method for operating an object recognition model configured to recognize a traffic light object, the method comprising: obtaining an input image from a camera included in a vehicle, the input image being among frames captured by the camera including a previous frame; Based on prior information about the traffic light object, estimating a first region of interest RoI for the input image; determining a second RoI based on the first RoI and based on a detection result of the previous frame, wherein the detection result corresponds to a recognition result of the traffic light in the previous frame being recognized by the object recognition model; and A traffic light in the input image is recognized by the object recognition model, wherein the recognition is based on the input image and the second RoI.

2. The operating method according to claim 1, wherein: Identifying the traffic light comprises: Obtaining a target image by extracting a portion of the input image corresponding to the second RoI; and The target image is enlarged to a size corresponding to the size of the input image, wherein recognition by the object recognition model is also based on the enlarged target image.

3. The operating method according to claim 1, wherein: The prior information about the traffic light object includes: Distribution data corresponding to a distribution of previously observed positions of traffic lights selected based on having a magnitude less than a threshold.

4. The operating method according to claim 3, wherein: The prior information about the traffic light object further includes: information related to the first RoI generated based on the distribution data.

5. The operating method according to claim 1, wherein: The prior information about the traffic light object includes driving environment information related to the environment in which the vehicle is driving and the specifications of the camera.

6. The operating method according to claim 1, wherein: Estimating the first RoI further includes adjusting the first RoI based on information from sensors of the vehicle.

7. The operating method according to claim 1, wherein: Estimating the first RoI further includes adjusting the first RoI based on specifications of the camera.

8. The operating method according to claim 1, wherein: Determining the second RoI includes: aggregating the detection results of the previous frames; obtaining a center of a cluster of the detection results of the previous frame; filtering the centers; and Based on the filtered center, the second RoI is obtained.

9. The operating method according to claim 8, wherein: Aggregating the detection results of the previous frames includes maintaining the detection results of a rolling window of the previous frames, the rolling window including a time window or a window defined by a plurality of previous frames.

10. The operating method according to claim 8, wherein: Determining the second RoI includes: obtaining a distance function between the centers; removing centers that do not satisfy the minimum condition in the distance function; and The second RoI is determined based on the center that meets the minimum condition.

11. An electronic device, comprising: camera; one or more processors; as well as A memory storing instructions, wherein the instructions are configured to cause the one or more processors to perform the following operations: obtaining an input image from the camera; Based on prior information about the traffic light object, estimating a first region of interest RoI for the input image; determining a second RoI based on the first RoI and based on a detection result of a previous frame of the camera, wherein the detection result of the previous frame corresponds to a recognition result of a traffic light in the previous frame by an object recognition model, the object recognition model being configured to recognize a traffic light; and A traffic light in the input image is recognized by the object recognition model, wherein the recognition is based on the input image and the second RoI.

12. The electronic device according to claim 11, wherein: The instructions are further configured to cause the one or more processors to: Obtaining a target image by extracting a portion of the input image corresponding to the second RoI; as well as enlarging the target image to a size corresponding to the size of the input image, and The recognition performed by the object recognition model is also based on the enlarged target image.

13. The electronic device according to claim 11, wherein: The prior information includes: distribution data corresponding to a distribution of previously observed positions of traffic lights selected based on having a size less than a threshold; and wherein the prior information further comprises information related to the first RoI generated based on the distribution data.

14. The electronic device according to claim 11, wherein: The electronic device is included in a vehicle, and wherein the instructions are further configured to cause the one or more processors to perform the following operations: adjust the first RoI based on sensors of the vehicle and according to a driving environment of the vehicle.

15. The electronic device according to claim 11, wherein: The instructions are further configured to cause the one or more processors to: aggregating the detection results of the previous frames in their time order, obtaining the center of the cluster of the detection results of the previous frame, and Some of the centers are removed, and the second RoI is determined based thereon.

16. The electronic device according to claim 15, wherein: The instructions are also configured to cause the one or more processors to perform the following operations: updating the detection result of the previous frame to include a result of identifying a traffic light in the input image, wherein the updated detection result is used to identify a traffic light in a next input image captured by the camera.

17. The electronic device according to claim 15, wherein: The instructions are further configured to cause the one or more processors to: Obtain the distance function between the centers, removing centers other than those satisfying a minimum condition in the distance function, and The second RoI is placed with respect to the remaining centers that satisfy the minimum condition.

18. A method performed by a computing device, the method comprising: receiving an input image captured by a camera of a moving vehicle; For a first input image among the input images, determining a first region of interest RoI, wherein the first RoI is determined based on historical traffic light observations; inputting the first input image and the enlarged image of the area defined by the first RoI in the first input image to an object recognition model configured to recognize a traffic light, wherein the object recognition model infers a position of a first traffic light in the first input image based on the first input image and the enlarged image; determining a second RoI for a second input image captured after the first input image in the input images, wherein the second RoI is determined based on a position of the first traffic light in the first input image; and The second input image and an enlarged image of an area defined by the second RoI in the second input image are input to the object recognition model, and the object recognition model infers a position of the first traffic light or the second traffic light in the second input image based on the second input image and the enlarged image.

19. The method according to claim 18, further comprising: The position of the second RoI is adjusted based on the first RoI or based on the historical traffic light observations.

20. The method according to claim 18, wherein: The object recognition model comprises a neural network, and wherein the inferred locations comprise corresponding bounding boxes.

Images