Apparatus for estimating position of object, and method therefor

Abstract

An apparatus for estimating the position of an object is presented. The apparatus comprises: an image sensor for acquiring an image of the surroundings of a road; and a processor for detecting an object from the acquired image, wherein the processor can determine whether a portion of a first bounding box of the detected object is positioned in a preset region in the acquired image or extends beyond the frame boundary of the acquired image, and update the first bounding box to a second bounding box according to the portion of the first bounding box being positioned in the preset region in the acquired image or extending beyond the acquired image.

Description

Device for estimating object position and method therefor

[0001] The present invention relates to a device and method for estimating an object location, and more specifically, to a technique for estimating an object location through object detection in an acquired image.

[0002] Figure 1 illustrates a traffic safety service system based on communication technology, one of the fields to which the present invention applies. As an example, it illustrates the use of V2X (vehicle to everything) and Soft V2X technologies.

[0003] The road side unit (RSU) (100) transmits road and traffic information provided by the Cooperative Intelligent Transport System (C-ITS) (200) to road users. Furthermore, the RSU (100) collects information on surrounding vehicles. Furthermore, the RSU (100) links C-ITS and Soft V2X services, which use different technologies, and utilizes information from various roadside sensors, including CCTV, to provide information that can predict road collisions.

[0004] In addition, the RSU (100) can detect various road users such as pedestrians, vehicles, two-wheeled vehicles, and kickboards using intelligent CCTV. The RSU (100) transmits road user detection information to surrounding users to enable collision prediction or avoidance. The RSU (100) can transmit road user detection information to a C-ITS terminal (311) and a Soft V2X terminal (321). The C-ITS terminal (311) is installed in a vehicle (310), and the Soft V2X terminal (321) is illustrated as being installed or located in a vehicle (320).

[0005] In this way, C-ITS and Soft V2X are technologies that help predict, avoid, or prevent traffic accidents, respectively, but C-ITS terminals (311) and Soft V2X terminals (321) cannot directly exchange information with each other. In this way, by linking C-ITS and Soft V2X through the RSU (100), information exchange between C-ITS terminals and Soft V2X terminals is enabled, thereby enabling collision prediction and avoidance between C-ITS terminal users and Soft V2X terminal users.

[0006] As described above, achieving collision prediction, avoidance, and other tasks requires, above all, the advancement of object location prediction, estimation, and tracking technologies. The present invention addresses technologies for detecting objects within an image and predicting, estimating, and tracking their locations.

[0007] The present invention proposes a method for more accurately estimating the location of an object when tracking an object in an acquired image.

[0008] More specifically, the present invention proposes a method for correcting or updating observation values ​​that serve as a basis for estimating the current state of an object.

[0009] The problems to be solved by the present invention are not limited to the problems to be solved above, and other problems not mentioned can be clearly understood by a person having ordinary skill in the technical field to which the present invention belongs from the description below.

[0010] A device for estimating an object location is proposed, the device including an image sensor for acquiring an image of the surroundings of a road; and a processor for performing object detection from the acquired image, wherein the processor is configured to determine whether a part of a first bounding box of the detected object is located in a preset area within the acquired image or a part of the first bounding box is outside a frame boundary of the acquired image, and update the first bounding box to a second bounding box according to whether a part of the first bounding box is located in a preset area within the acquired image or a part of the first bounding box is outside the acquired image.

[0011] A method for object position estimation is proposed, the method comprising: performing object detection in an acquired image around a road; determining whether a portion of a first bounding box of the detected object is located in a preset region within the acquired image or whether a portion of the first bounding box is outside a frame boundary of the acquired image; and updating the first bounding box to a second bounding box depending on whether a portion of the first bounding box is located in a preset region within the acquired image or a portion of the first bounding box is outside the acquired image.

[0012] The above problem solving methods are only some of the embodiments of the present invention, and various embodiments reflecting the technical features of the present invention can be derived and understood by a person having ordinary knowledge in the relevant technical field based on the detailed description of the present invention described below.

[0013] The present invention has the following effects.

[0014] The present invention can correct or update observation values ​​in tracking objects to values ​​that better reflect the actual location of the object.

[0015] The present invention can improve the accuracy of object location estimation in object tracking.

[0016] The effects according to the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art to which the present invention pertains from the detailed description of the invention below.

[0017] The accompanying drawings, which are included as part of the detailed description to aid in understanding the present invention, provide embodiments of the present invention and, together with the detailed description, explain the technical idea of ​​the present invention.

[0018] Figure 1 illustrates a system diagram of a field to which the present invention is applied.

[0019] Figure 2 illustrates a flowchart for camera-based object tracking and position estimation to which the present invention is applied.

[0020] Figure 3 illustrates the procedure for object tracking using a Kalman filter.

[0021] Figures 4 and 5 show images of the road surroundings.

[0022] Figures 6 and 7 describe problems and improvements for cases where an object enters a preset area in an acquired image and is obscured.

[0023] Figure 8 describes the problem and improvement for cases where an object deviates from the acquired image.

[0024] FIG. 9 is a diagram illustrating a method for correcting or updating a bounding box according to the present invention.

[0025] Figure 10 illustrates a procedure for object tracking according to the present invention.

[0026] Figure 11 shows an image obtained according to the present invention.

[0027] Figure 12 illustrates a flowchart for object tracking or object status update according to the present invention.

[0028] Fig. 13 illustrates a block diagram of an object position estimation device according to the present invention.

[0029] Hereinafter, embodiments disclosed in this specification will be described in detail with reference to the attached drawings. Regardless of the drawing numbers, identical or similar components will be given the same reference numbers and redundant descriptions thereof will be omitted. The suffixes "module" and "part" used for components in the following description are assigned or used interchangeably only for the convenience of writing the specification, and do not in themselves have distinct meanings or roles. In addition, when describing the embodiments disclosed in this specification, if it is determined that a specific description of a related known technology may obscure the gist of the embodiments disclosed in this specification, a detailed description thereof will be omitted. In addition, the attached drawings are only intended to facilitate easy understanding of the embodiments disclosed in this specification, and the technical ideas disclosed in this specification are not limited by the attached drawings, and should be understood to include all modifications, equivalents, and substitutes included in the spirit and technical scope of the present invention.

[0030] Terms that include ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by these terms. These terms are used solely to distinguish one component from another.

[0031] When a component is referred to as being "connected" or "connected" to another component, it should be understood that it may be directly connected or connected to that other component, but that there may be other components intervening. Conversely, when a component is referred to as being "directly connected" or "connected" to another component, it should be understood that there are no other components intervening.

[0032] Singular expressions include plural expressions unless the context clearly indicates otherwise.

[0033] In this application, terms such as “include” or “have” are intended to specify the presence of a feature, number, step, operation, component, part or combination thereof described in the specification, but should be understood not to exclude in advance the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts or combinations thereof.

[0034]

[0035] Figure 2 illustrates a procedure in a system for camera-based object tracking and position estimation to which the present invention is applied.

[0036] First, an image is acquired using an image sensor such as a camera (S10).

[0037] Object detection is performed in the acquired image (S20).

[0038] Then, tracking of the detected object is performed (S30).

[0039] Finally, location estimation is performed to obtain location information such as GPS coordinates from the location of the object in the acquired image (S40).

[0040] Object tracking uses computer vision technology to detect and track objects in images. To achieve this, various algorithms are used to detect and track moving objects. Key algorithms include the Kalman filter, particle filter, and DeepSORT. Each tracked object is assigned an identifier (ID) and tracked individually. This allows the path and speed of each object to be estimated.

[0041]

[0042] Figure 3 illustrates the procedure for object tracking using a Kalman filter.

[0043] The Kalman filter is used to track objects within acquired images, and is largely composed of a prediction step and an update step. By repeating this, the position of the object can be continuously tracked.

[0044] Object tracking illustrated in Fig. 3 can be expressed as follows.

[0045]

[0046] x_c, y_c represent the current state (optimal state) of the object or the value of the current state (optimal state) at time = tT, and represent the x-axis midpoint and y-axis midpoint of the bounding box or the object, respectively. Here, T represents the cycle in which object tracking is repeated, such as obtaining the predicted state of the object position (prediction), obtaining the observed value of the object position (measurement) (observation), and obtaining the current state value of the object (update).

[0047] x_c' and y_c' represent the predicted state or predicted state value of the object at time = t, respectively, and represent the x-axis midpoint and y-axis midpoint of the bounding box or the object, respectively.

[0048] The current state or current state value of an object at time = tT is determined based on the predicted state value and the position observation value at time = tT. For example, the current state value of an object at time = tT is determined as the average of the predicted state value of the object at time = tT and the position observation value of the object.

[0049] Additionally, the predicted state value of the object at time = t is determined based on the current state value of the object at time = tT. That is, the predicted state value of the object is determined based on the current state value of the previous object.

[0050] The predicted position state value of the object, the observed position value of the object, and the current state value of the object are values ​​expressed and obtained as bounding box or center point information of the object.

[0051] Object tracking according to Figure 3 is performed under the assumption that the bounding box size does not change, change, or update. However, using the width and height of the bounding box for object tracking, rather than solely using the object's "position" (i.e., the coordinates of its center point), may improve accuracy.

[0052] That is, based on FIG. 3 and mathematical expression 1, object tracking according to the present invention can be expressed as follows.

[0053]

[0054] Here, w is the width of the bounding box of the current state at time = tT, h is the height of the bounding box of the current state at time = tT, w' is the width of the bounding box of the position prediction state or the position prediction state value at time = t, and h' is the height of the bounding box of the position prediction state or the position prediction state value at time = t. In this case, the position prediction state value of the object, the position observation value of the object, and the current state value of the object are values ​​expressed and obtained by the bounding box or the center point information of the object and the width and height of the bounding box.

[0055]

[0056] Let us explain the procedure in Figure 3 in more detail.

[0057] A Kalman filter can be used to obtain an object position prediction state or its value (S31). In a Kalman filter, a state can be expressed as a vector containing the position and velocity of the object to be tracked.

[0058] The predicted state of the object position at time t is determined using the current state of the object at time tT (hereinafter referred to as the “optimal state”) and the system model. Here, T corresponds to the cycle in which the object position prediction is repeated. The optimal state of the object at time tT is updated based on the predicted state of the object at time tT and the observed position of the object. Initially, there is no estimated state for the object, so an initial state estimate can be set.

[0059] To update the optimal state value of an object, the position observation value of the object is acquired from the acquired image (S32). The position observation value of the object is the actually measured position of the object, and is the position information of the object acquired through an image processing technique. The position observation value of the object may include the center coordinates of the bounding box or the center coordinates of the object (or the bounding box) (see Equation 1). In addition, the position observation value of the object may include the width and height of the bounding box (see Equation 2).

[0060] As mentioned above, the optimal state value of the new object is updated using the predicted state value of the object's position and the observed state value of the object's position (S33).

[0061] The above-described procedure can be repeated after the update. During this process, the Kalman gain is determined using the error and / or error covariance matrix between the predicted state value and the observed position value, and the Kalman gain is used as a weight to adjust the error between the predicted state value and the observed position value.

[0062] As previously explained, object position observation values ​​include the center coordinates of the bounding box, or the center coordinates of the object (or bounding box), and the width and height of the bounding box. However, in the scenarios illustrated in Figures 4 and 5, the object is obscured by a specific object, making the current bounding box size smaller than that of the previous tracking point, resulting in an error being reflected in the object position observation values. This will be described in more detail with reference to Figures 4 and 5.

[0063]

[0064] Figures 4 and 5 illustrate images (1000) of the road surroundings. It is assumed that Figure 4 is an image captured at time = t, and Figure 5 is an image captured at time = t + T.

[0065] A scene is shown in which vehicles (310, 320, 330) are driving on a road, and bounding boxes (B1, B2, B3) are created for each.

[0066] The video (1000) around the road is obtained through CCTV (Closed Circuit Television) installed around the road.

[0067] Referring to FIG. 5, it is illustrated that the vehicle (310) is covered by an area (R1) within the acquired image (1000). This is because there are objects (illustrated as road signs and traffic light facilities) closer than the vehicle (310) on the line of sight between the CCTV and the subject (i.e., the road surroundings). Accordingly, the bounding box (B1) for the vehicle (310) within the acquired image (1000) illustrated in FIG. 5 is smaller in size than the bounding box (B1) for the vehicle (310) illustrated in FIG. 4. As a result, even though time has passed from t to t+T and the vehicle (310) has actually been traveling in the traveling direction, the position of the vehicle determined by the center coordinates of the bounding box may not change, or an error may occur, such as an error in which the vehicle has moved in the opposite direction of the traveling direction. This is reflected as an error in the object position observation value in the object tracking procedure using the Kalman filter described above, and this leads to an error in the optimal state value of the object (S33 in Fig. 3).

[0068] In addition, in the situation shown in Fig. 5, a phenomenon occurs in which the bounding box (B1) is not fixed and shakes, which ultimately causes an error in the object position observation value.

[0069] Additionally, errors in object position observation values ​​may occur not only when the object is located in a specific area within the acquired image (1000) and at least part of it is obscured, but also when the object leaves the coverage area of ​​the acquired image (1000), i.e., when part of the object no longer exists in the acquired image (1000).

[0070]

[0071] Figure 6 describes a problem and improvement thereof when an object enters a preset area in an acquired image and is obscured.

[0072] Referring to (a) of Fig. 6, a preset region (R1) is set in the acquired image (1000). A bounding box (B) of an object such as a vehicle is created and displayed, and since at least a portion of the object is obscured in the preset region (R1), the bounding box (B) can be created and displayed as a solid line.

[0073] However, since the location of the actual object is the location including the dotted line, according to the present invention, as shown in (b) of FIG. 6, the size of the bounding box (B_u) is corrected or updated, and thus the center point (P_c,u) of the bounding box or object is determined according to the size and location of the corrected or updated bounding box (B_u).

[0074] Figure 6 (c) shows the bounding box (B) obtained in Figure 6 (a) without updating the bounding box. In this case, the center point (P_c) of the bounding box or object is determined based on the size and position of the bounding box (B).

[0075] Referring to (b) and (c) of Fig. 6, it can be confirmed that each center point (P_c, P_c,u) has a difference, as explained above with reference to Fig. 5. This leads to a difference in the object position observation value, which is ultimately reflected as an error in the object position prediction value or the object's optimal state value.

[0076] Figure 7 illustrates a detected bounding box and its correction or update, which is different from that of Figure 6 (a).

[0077] Referring to (a) of Fig. 7, a preset region (R1) is set in the acquired image (1000). A bounding box (B) of an object such as a vehicle is generated and displayed, and since at least a portion of the object is obscured in the preset region (R1), the bounding box (B) can be generated and displayed as a solid line. Unlike (a) of Fig. 6, (a) of Fig. 7 illustrates a case where a portion of the bounding box (B) is located in the preset region (R1).

[0078] Figure 7 (c) shows the bounding box (B) obtained in Figure 7 (a) without updating the bounding box. In this case, the center point (P_c) of the bounding box or object is determined based on the size and position of the bounding box (B).

[0079] Figure 7 (b) shows a bounding box (B_u) corrected or updated according to the present invention.

[0080] In this way, the present invention seeks to correct or update the bounding box (B_u) as in (b) of Fig. 6 or (b) of Fig. 7 when the detected bounding box (B) is adjacent to or located in a preset area (R1) of the acquired image, as in (a) of Fig. 6 or (a) of Fig. 7.

[0081] Furthermore, more broadly, the bounding box (B) can be corrected or updated even if it is within a predetermined distance or range from the region (R1). In other words, the bounding box (B) is corrected or updated even if it is not actually located within the predetermined region (R1).

[0082] Whether the detected bounding box (B) is located in the preset region (R1) can be determined based on position information, such as coordinate information, of the bounding box (B) and the preset region (R1). In the present invention, the position information of the bounding box (B) in the acquired image and the position information of the preset region (R1) correspond to information that can be acquired through image processing. In addition, the bounding box (B) contacting the preset region (R1) means that some of the coordinates of the boundary of the bounding box (B) are the same as the coordinates of the boundary of the preset region (R1), while the coordinates of the entire boundary of the bounding box (B) are not the same as the coordinates inside the boundary of the preset region (R1).

[0083] Meanwhile, in Fig. 6 (b), Fig. 7 (a), Fig. 7 (b), and Fig. 7 (c), the bounding box (B, B_u) is depicted over the preset region (R1), but in the acquired image, R1 and the bounding box may be displayed in an overlapping state.

[0084]

[0085] Figure 8 describes the problem and improvement for cases where an object deviates from the acquired image.

[0086] Referring to (a) of Fig. 8, a portion of the bounding box (B) of the object is shown to be out of range in the acquired image (1000). This indicates a situation where a portion of the object detected in the acquired image is out of the coverage area being captured by an image sensor such as a CCTV.

[0087] A bounding box (B) of an object such as a vehicle is generated and displayed, and since at least a portion of the object is outside the coverage area, the bounding box (B) can be generated and displayed only as a solid line portion. That is, according to a conventional technique, the bounding box (B) can be set as in (c) of Fig. 8. In this case, the center point (P_c) of the bounding box or object is determined according to the size and position of the bounding box (B).

[0088] According to the present invention, as shown in (b) of FIG. 8, the size of the bounding box (B_u) is corrected or updated, and thereby the center point (P_c,u) of the bounding box or object is determined according to the size and position of the corrected or updated bounding box (B_u).

[0089] Referring to (b) and (c) of Fig. 8, it can be confirmed that each center point (P_c, P_c,u) has a difference, as explained above with reference to Fig. 5. This leads to a difference in the object position observation value, which is ultimately reflected as an error in the object position prediction value or the object's optimal state value.

[0090] In this way, the present invention seeks to correct or update the bounding box (B_u) as in (b) of Fig. 8 when the detected bounding box (B) deviates from the boundary of the acquired image, as in (a) of Fig. 8.

[0091] Here, the bounding box (B) deviating from the boundary of the acquired image includes the case where the bounding box (B) touches the boundary of the acquired image. Furthermore, to expand further, the bounding box (B) deviating from the boundary of the acquired image also includes the case where the bounding box (B) is within a predetermined distance or range from the boundary of the acquired image. In other words, the case where the bounding box (B) is located within a predetermined distance or range without actually deviating from the boundary of the acquired image is also included.

[0092] Whether the detected bounding box (B) exceeds the boundary of the acquired image can be determined based on the bounding box (B) and the size or boundary information of the acquired image, such as coordinate information. In the present invention, the position information of the bounding box (B) in the acquired image and the size or boundary information of the acquired image correspond to information that can be acquired through image processing. In addition, the bounding box (B) touching the boundary of the acquired image means that some of the boundaries of the bounding box (B) are the same as the coordinates of the boundary of the acquired image.

[0093]

[0094] FIG. 9 is a diagram illustrating a method for correcting or updating a bounding box according to the present invention.

[0095] A preset region (R1) may be set in the acquired image (1000). When an object is positioned in the preset region (R1), the overlapping portion between the preset region and the object is hidden in the acquired image (1000). The shape, size, position, number, etc. of the preset region (R1) of the acquired image (1000) are merely examples and do not limit the present invention.

[0096] A device (e.g., an object position estimation device) according to the present invention can generate and display bounding boxes (B11, B12, B13, B14) of an object in an acquired image (1000). The bounding boxes (B11, B12, B13, B14) illustrated in FIG. 9 are bounding boxes generated for measured or detected objects, and are bounding boxes before being corrected or updated.

[0097] The illustrated bounding boxes are four in number, but this is for illustrative purposes only and does not limit the scope of the present invention. Furthermore, it is assumed that all illustrated bounding boxes are rectangles of the same size.

[0098] The bounding box information can be composed of position information and size information. Assuming that the upper left edge of the acquired image (1000) is the origin, the bounding box information can be expressed as the minimum distance (b_box.top, b_box.bottom, b_box.left, b_box.right) from two straight lines (or two axes) (i.e., the x-axis and the y-axis) forming the origin to each side of the bounding box corresponding to it. In addition, the combination of the minimum distances to each side ultimately corresponds to the coordinate information of each edge. That is, assuming that the bounding box is a rectangle, if b_box.top, b_box.bottom, b_box.left, b_box.right are specified, the position and size information of the bounding box can be specified.

[0099] At this time, b_box.top represents the minimum distance between the horizontal line passing through the origin (i.e., the x-axis) and the upper side of the bounding box, b_box.bottom represents the minimum distance between the horizontal line passing through the origin and the lower side of the bounding box, b_box.left represents the minimum distance between the vertical line passing through the origin (i.e., the y-axis) and the left side of the bounding box, and b_box.right represents the minimum distance between the vertical line passing through the origin and the right side of the bounding box.

[0100] In addition, the bounding box information can be expressed as the minimum distance to the two straight lines forming the origin and the two corresponding perpendicular sides, and the width and height of the bounding box. In other words, the position and size information of the bounding box can be expressed with the coordinate information of one of the edges and the width and height of the bounding box. The position and size of the bounding box can be expressed using either b_box.top or b_box.bottom of the bounding box, either b_box.left or b_box.right, and the width and height of the bounding box. In other words, the position and size of the bounding box can be expressed using the coordinates (relative to the origin) of one of the edges of the bounding box and the width and height of the bounding box.

[0101] Meanwhile, since it is difficult or impossible to detect the width and height of the bounding box in the acquired image (1000) when the object is occluded in the preset region (R1), the width and height detected or acquired in the already acquired image may be used. Preferably, the width and height of the bounding box detected or acquired in the image in which the object's bounding box is not occluded in the preset region (R1) may be used.

[0102] Based on the above explanation, if one of b_box.top and b_box.bottom and one of b_box.left and b_box.right (ultimately, the coordinate information of the edge) of one edge of the bounding box (B11, B12, B13, B14) that is not covered by a preset area (R1) is used, the bounding box can be corrected or updated by utilizing the width and height of the bounding box that has been previously obtained. That is, a corrected or updated bounding box is obtained by utilizing the coordinate information of the edge of the bounding box that is not covered by R1 and the width and height of the bounding box that has been previously obtained.

[0103] However, it's important to determine which edge is used as a reference for correcting or updating the bounding box. This is because the bounding box position resulting from correction or update varies depending on the reference edge, which in turn affects the observed object position. This ultimately means that the predicted object position or the optimal state of the object will vary.

[0104] For example, let's explain B11. It has an upper left edge and an upper right edge that are not covered by R1. Assume that the coordinates of the upper left edge are (3, 1) and the coordinates of the upper right edge are (5, 1). Assume that the width of the previously acquired bounding box is 3 and the height is 7. In this case, if the bounding box is updated using the upper left edge as the reference edge, the coordinates of the lower right edge of the bounding box become (6, 8). If the bounding box is updated using the upper right edge as the reference edge, the coordinates of the lower left edge of the bounding box become (2, 8). In this way, the position of the corrected or updated bounding box changes depending on the reference edge.

[0105] Therefore, it is necessary to determine a reference edge to be used for correction or updating the bounding box. It is preferable that the reference edge be the edge opposite the edge that penetrates most deeply into the preset region (R1).

[0106] The procedure for correcting or updating the bounding box is briefly summarized as follows.

[0107] a) Bounding box detection

[0108] b-1) Determine the edge located in the preset area (R1) among the detected bounding boxes.

[0109] b-2) When multiple edges are located in R1, select a specific edge (the edge that is located deepest in R1).

[0110] c) Determine the edge opposite or diagonal to the selected edge as the reference edge.

[0111] d) Correct or update the bounding box based on the reference edge.

[0112] Among these, except for process b-2), everything has already been explained, so we will explain b-2). One edge selected in b-2) is referred to as the “base edge.”

[0113] Assume that the edges located in the preset region (R1) have been determined. Referring to Fig. 9, in the case of the bounding box (B11), the two lower edges (i.e., the lower left edge and the lower right edge) are located in the preset region (R1).

[0114] For each of the two edges on the lower side, a side or line (or straight line) connected or formed with a neighboring edge that is not located in the preset region (R1) is obtained. Then, for the obtained side or line, a length located in the preset region (R1) is obtained. The edge of the side or straight line with the longest length is selected as the base edge.

[0115] The process of obtaining a reference edge or a base edge, especially b-2), is a process necessary when two edges of the first bounding box are located in a preset area (R1), assuming that the bounding box is a rectangle. If only one edge of the rectangle is located in the preset area (R1), that edge is determined to be the base edge, and an edge located diagonally from the base edge can be determined as the reference edge. If only three edges of the rectangle are located in the preset area (R1), there is only one edge that is not located in the preset area (R1), so that edge can be determined as the reference edge.

[0116] Even if the first bounding box touches the preset region (R1), if one of the edges of the first bounding box touches R1, the edge will be determined as the base edge. If two edges of the first bounding box touch the preset region (R1), one of the two edges can be determined as the base edge. An edge located diagonally from the determined base edge can be determined as the reference edge. If three edges of the first bounding box touch the preset region (R1), an edge that does not touch the preset region (R1) can be determined as the reference edge.

[0117] In the case of the bounding box (B11) in the upper left of Fig. 9, the two lower edges are located in a preset area (R1). The object position estimation device detects the bounding box (B11) from the acquired image (a). The object position estimation device determines two edges of the bounding box (B11) located in the preset area (R1) (b-1). Then, the object position estimation device selects P_r1 among the two edges as the base edge (b-2). The selection process refers to the above-described content. The object position estimation device determines an edge located on the diagonal of the base edge or not adjacent to it as the base edge (c). The object position estimation device can perform correction or update of the bounding box using the coordinate information of the base edge and the width and height information of the previously acquired bounding box (d). The width and height information of the obtained bounding box are values ​​obtained from a previous object position estimation, and preferably include width and height information of the bounding box obtained in a state where the object is not covered by an area such as a preset area (R1).

[0118] For the bounding box (B11), the upper left edge is taken as the reference, and accordingly, the minimum distance between each side and the reference axis can be determined as follows. In this specification, the minimum distance can be expressed in units of coordinate values.

[0119] B_box.left = b_box.left

[0120] B_box.top = b_box.top

[0121] B_box.right = b_box.left + pre_b.box.width

[0122] B_box.bottom = b_box.top + pre_b.box.height

[0123] “b_box” refers to the minimum distance between the corresponding side of the bounding box detected from the acquired image and the reference axis, and b_box.right refers to the minimum distance between the right side of the bounding box and the reference axis (the vertical line passing through the origin).

[0124] “B_box” refers to the minimum distance between the corresponding edge of the corrected or updated bounding box and the reference axis, and B_box.right refers to the minimum distance between the right edge of the corrected or updated bounding box and the reference axis (i.e., the y-axis) (ultimately, the x-coordinate value of the upper right or lower right edge). For the bounding box (B11), the information on the right and lower edges was obtained using the information on the upper and left edges (i.e., the minimum distance from each axis) and the width and height of the bounding box.

[0125] To explain this again, if we know the coordinates (b_box.left, b_box.top) of the upper left edge of the bounding box that is not located in the region (R1), we can obtain information about the updated bounding box using the width and height information of the previously obtained bounding box.

[0126] For the bounding box (B12) in the upper right, the lower right edge becomes the base edge (P_r2), and accordingly, the upper right edge becomes the reference edge. Therefore, the minimum distance between each side and the reference axis can be determined as follows.

[0127] B_box.right = b_box.right

[0128] B_box.top = b_box.top

[0129] B_box.left = b_box.right - pre_b.box.width

[0130] B_box.bottom = b_box.top + pre_b.box.height

[0131] For the bounding box (B13) at the lower left, the upper right edge (P_r3) becomes the base edge, and the lower left edge becomes the reference edge, in the same manner as the correction or update method for the bounding boxes (B11, B12) described above. The minimum distance between each side and the reference axis can be determined as follows.

[0132] B_box.left = b_box.left

[0133] B_box.bottom = b_box.bottom

[0134] B_box.right = b_box.left + pre_b.box.width

[0135] B_box.top = b_box.bottom - pre_b.box.height

[0136] For the bounding box (B14) at the lower right, the upper left edge (P_r4) becomes the base edge, and the lower right edge becomes the reference edge, in the same manner as the correction or update method for the bounding boxes (B11, B12) described above. The minimum distance between each side and the reference axis can be determined as follows.

[0137] B_box.right = b_box.right

[0138] B_box.bottom = b_box.bottom

[0139] B_box.left = b_box.right - pre_b.box.width

[0140] B_box.top = b_box.bottom - pre_b.box.height

[0141] Meanwhile, the method for indicating the position of the bounding box may also utilize other methods, such as the relative coordinates of each edge of the bounding box from the origin. The present invention is not limited by the method for indicating the position of the bounding box.

[0142]

[0143] Meanwhile, FIG. 9 did not provide an explanation for the case where some of the first bounding boxes are outside the frame boundaries of the acquired image, or where some of the objects are outside the frame boundaries of the acquired image (i.e., only some of the objects are within the frame boundaries of the acquired image).

[0144] Let us explain the update to the second bounding box when the first bounding box or part of the object moves outside the frame boundaries of the acquired image.

[0145] In this case, the procedure for correcting or updating the bounding box is as follows.

[0146] e) Bounding box detection or detection of whether part of an object is outside the frame boundaries of the acquired image.

[0147] f) Among the detected bounding boxes, one of the edges within the frame boundary of the acquired image is determined as the reference edge.

[0148] g) Correct or update the bounding box based on the reference edge.

[0149]

[0150] Assuming that the first bounding box is rectangular and does not rotate within the acquired image, the first bounding box will always have two adjacent edges that extend beyond the boundaries of the acquired image. Therefore, the processing is slightly different from the case where the edges of the first bounding box are located within a preset region (R1). This will be explained below.

[0151] If a part of the detected object goes beyond the frame boundary of the acquired image, the bounding box can be detected and generated only within the frame boundary. Therefore, the object position estimation device can determine whether a part of the object has gone beyond the boundary of the acquired image by determining whether the bounding box contacts the frame boundary or whether a bounding box smaller than the size of the previously acquired bounding box has been acquired (e). Thereafter, the object estimation device can select one of the two edges of the bounding box within the boundary of the acquired image as a reference edge. If the object estimation device can acquire the movement direction of the object over time, the acquired movement direction can be used to select the reference edge. Preferably, the edge located further back in the movement direction can be the reference edge. For example, referring to the first bounding box detected in the acquired image, if the first bounding box moves in a diagonal direction downward to the right and goes beyond the frame boundary of the acquired image, the upper left edge based on the acquired image can be the reference edge.

[0152] However, when explaining based on the first bounding box detected in the acquired image, if the first bounding box moves only in the horizontal direction (left or right) or vertical direction (up or down), any one of the two edges can become the reference edge.

[0153] Meanwhile, correction or update of the bounding box according to the present invention may be performed even when at least a portion of the detected bounding box (B11, B12, B13, B14) touches a preset area (R1) within the acquired image (1000). Additionally or alternatively, correction or update of the bounding box according to the present invention may be performed even when the bounding box is located at a predetermined distance or range from the area (R1).

[0154]

[0155] Figure 10 illustrates a procedure for object tracking according to the present invention.

[0156] The procedure for object tracking illustrated in Fig. 10 is based on the object tracking of Fig. 3. Therefore, description of the same parts as in Fig. 3 will be omitted.

[0157] As previously described, object tracking according to the present invention includes correction or updating of the bounding box (S92-1). Furthermore, additional information required for object tracking is the width and height of the bounding box in the current or predicted state of the object. Therefore, the present invention utilizes object tracking according to mathematical equation (3).

[0158] The correction or update of the bounding box (S92-1) may be performed when at least a part of the bounding box of the object is located in, touches, or is located at a predetermined distance or range from a preset area within the acquired image, or when the bounding box of the object touches the boundary of the acquired image.

[0159] That is, the technique proposed in this specification enables the prediction state value of an object's position, the observation value of an object's position, and the current state value of an object to be expressed or obtained as the width and height of an updated bounding box as well as the bounding box or the center point information of the object.

[0160]

[0161] Object tracking according to the present invention includes updating the bounding box of the object or the observation values ​​related thereto, so that the accuracy of object position tracking can be guaranteed even if a part of the object is obscured or disappears in the acquired image (1000).

[0162]

[0163] Fig. 11 illustrates an acquired image (1001) according to the present invention. As described above, it can be seen that the size or position of the bounding box (B1) is maintained even when the object (310) is obscured by the preset region (R1), as a result of the correction or update of the bounding box (B1). In Fig. 5, the object is obscured by the preset region (R1), and thus the size of the bounding box is generated and displayed only for the portion not obscured by the preset region (R1).

[0164]

[0165] Figure 12 illustrates a flowchart for object tracking or object status updating according to the present invention. Object tracking or object status updating may be performed by an object position estimation device or a processor included therein. For simplicity, the procedure will be described as being performed by the object position estimation device.

[0166] The procedure of Fig. 12 starts from a state in which a position prediction state value of an object detected in an image previously acquired by an object position estimation device is acquired. Object tracking is performed by sequentially repeating, as described above, acquisition of an (initial) object position prediction state value, acquisition of an object position observation value, and acquisition of an object current state value based on the object position prediction state value and the object position observation value. In Fig. 12, the focus is on acquisition of an object position observation value and acquisition of an object current state value using the result.

[0167] The object position estimation device can perform object detection in the acquired image (S1010).

[0168] The object position estimation device can attempt to detect whether the detected object is located in a preset area or out of the coverage area of ​​the acquired image (S1020).

[0169] Detecting whether the detected object is located in a preset area or out of the coverage area of ​​the acquired image is used to determine whether an update of the bounding box is required.

[0170] If an update of the bounding box is required (i.e., if the detected object is located in a preset area or is outside the coverage area of ​​the acquired image), the object position estimation device may correct or update the bounding box (S1040). The correction or update of the bounding box may be based on the content described above in FIGS. 6 to 9.

[0171] Then, the object position estimation device can determine a position observation value of the detected object based on the corrected or updated bounding box (hereinafter referred to as “position observation value based on updated bounding box”).

[0172] The object position estimation device can update the current state value (or optimal state value) of the object using the position observation value based on the updated bounding box (S1060). Updating the current state value of the object includes obtaining the current state value of the object based on the position observation value based on the updated bounding box and the most recent predicted state value of the object. For example, an average value of the position observation value based on the updated bounding box and the most recent predicted state value of the object can be determined as the current state value of the object.

[0173] If an update of the bounding box is not required (i.e., the detected object is not located in a preset area or does not go out of the coverage area of ​​the acquired image), the object position estimation device can update the current state value (or optimal state value) of the object using the position observation value based on the bounding box that has not been updated (S1050).

[0174] Although not shown, the object position estimation device can obtain the predicted position state value of the next object using the current state value of the updated object.

[0175]

[0176] Fig. 13 shows a block diagram of an object position estimation device (10) according to the present invention.

[0177] The object position estimation device (10) may include a transmitter / receiver (11) for transmitting and receiving messages or data; an image sensor (12) for acquiring images of the road surroundings; and a processor (13) for processing received messages or data, transmitting processed messages or data, or processing acquired images.

[0178] Depending on the embodiment, the object position estimation device (10) may include only one of the transceiver (11) and the image sensor (12). That is, the object position estimation device (10) may include the transceiver (11) and the processor (13), or may include the image sensor (12) and the processor (13).

[0179] The processor (13) can determine whether an object detected in the acquired image is located in a preset area within the acquired image, or whether a portion of the object is outside the boundaries of the acquired image or the frame of the acquired image. This is to determine whether correction or updating of the bounding box of the detected object is necessary.

[0180] Alternatively, the processor (13) may detect whether at least a portion of the first bounding box of the object detected in the acquired image is located within a preset region within the acquired image. This may be determined by determining whether one of the edges of the first bounding box is located within the preset region. In this case, it may be understood that the first bounding box contacting the preset region also includes the first bounding box being located within the preset region.

[0181] Alternatively, the processor (13) may compare the size of the bounding box of the object in the acquired image with the size of the bounding box previously acquired. By comparing the sizes of the bounding boxes, the necessity of correction or updating the bounding box of the detected object can be determined.

[0182] As correction or updating of the bounding box is required, the processor (13) may be configured to update the first bounding box of the detected object in the acquired image to a second bounding box.

[0183] The first bounding box corresponds to a shape for specifying the location of an object in an image acquired by an object detection model or algorithm, etc. The first bounding box is a mark that is automatically generated without considering whether the detected object is at least partially occluded by a preset region (R1) of FIG. 6, 7, or FIG. 9, or whether a part of the detected object is outside the acquired image as in FIG. 8. In the present invention, in order to increase the accuracy of object location estimation in the above cases, a second bounding box or an update to the second bounding box is proposed.

[0184] Here, the preset area may include an area in which an object detected in the acquired image is obscured by another fixed object, so that at least a portion of the detected object is not detected.

[0185] The processor (13) can generate a second bounding box of the detected object based on information about the width and height of the bounding box of the detected object that has been acquired in advance and information about the first bounding box. Here, the information about the first bounding box includes position information of at least one edge of the first bounding box (i.e., a two-dimensional coordinate value based on a specific point (origin) within the acquired image), and at least one edge must not be located in a preset area (R1) or must not be outside the acquired image.

[0186] Depending on the embodiment, the processor (13) may not display the first bounding box on the acquired image.

[0187] The generation of the second bounding box may include obtaining coordinates of the second bounding box of the object located in the preset area using the width and height of the bounding box of the object obtained in advance based on the reference edge of the first bounding box of the object not located in the preset area (R1).

[0188] Here, the reference edge may be located diagonally from one of the edges of the first bounding box located in the preset area. The selected edge of the first bounding box may include an edge whose line connecting the edge of the first bounding box located in the preset area with a neighboring edge that is not located in the preset area has the longest length and belongs to the preset area. P_r1, P_r2, P_r3, P_r4, etc., as shown in Fig. 9, correspond to the selected edges that determine the reference edge.

[0189] Alternatively, the generation of the second bounding box may include obtaining coordinates of the second bounding box using the width and height of the bounding box of the object obtained in advance, based on the reference edge of the first bounding box located within the frame boundary of the acquired image.

[0190] The midpoint of the second bounding box may be different from the midpoint of the first bounding box. Alternatively, at least one of the four edges of the second bounding box may be different from the four edges of the first bounding box.

[0191] The processor (13) may be configured to determine a current state value of the object by combining the position observation value of the object based on information about the second bounding box and the most recent position prediction state value of the object. The current state value of the object may be used to determine the next position prediction state value.

[0192] The processor (13) can determine the current state value of the object based on information about the center point coordinates and width and height of the generated second bounding box.

[0193] Additionally, the processor (13) may be configured to transmit a traffic safety-related message to a user or user terminal that is close within a preset distance from at least one of a position corresponding to the second bounding box, a position corresponding to an object position observation value based on the second bounding box, or a position corresponding to a position prediction status value based on the second bounding box via the transceiver (11).

[0194] According to the present invention, since more accurate object position observation values ​​or position prediction status values ​​can be obtained through updating the bounding box (i.e., the second bounding box), the selection of users or user terminals that need to receive traffic safety-related messages can be made more precise. For example, referring to Fig. 6, if, in the conventional method, users spaced apart by a certain distance from the center point (P_c) of the bounding box (B) of Fig. 6 (c) were the transmission targets of traffic safety-related messages, according to the present invention, users spaced apart by a certain distance from the center point (P_c,u) of the bounding box (B_u) of Fig. 6 (b) become the transmission targets of traffic safety-related messages. That is, a difference corresponding to the area corresponding to the distance difference between the two center points (P_c, P_c,u) occurs, and accordingly, users who could not receive traffic safety-related messages according to the conventional method can receive traffic safety-related messages according to the present invention. According to the present invention, in an emergency situation (e.g., a vehicle in a bounding box is approaching a crosswalk and a user is crossing the crosswalk), if the bounding box is partially obscured by a preset area (R1), a traffic-related message can be transmitted to the user.

[0195] A traffic safety-related message may include information about the location of the object. The information about the object's location may include at least one of the following: a predicted position value, an observed position value, or a current state value of the object, or an average of these values. Alternatively, the traffic safety-related message may further include, in addition to information about the object's location, at least one of the object's moving speed or moving direction.

[0196] Additionally, the processor (13) may be configured to transmit information related to a location corresponding to the second bounding box or a location corresponding to a location prediction status value based on the second bounding box to another server, such as a C-ITS server, via the transceiver (11).

[0197] Meanwhile, operations, processes, etc. described with reference to FIGS. 2 to 12, which are not described with reference to FIG. 13, may be performed by the processor (13).

[0198]

[0199] In addition, as another aspect of the present invention, the operation of the proposal or invention described above may be implemented, performed or executed by a “computer” (a comprehensive concept including a system on chip (SoC) or a (micro) processor, etc.), or may be provided as a code or a computer-readable storage medium storing or including the code or a computer program product, and the scope of the present invention may be extended to the code or the computer-readable storage medium storing or including the code or the computer program product.

[0200]

[0201] The detailed description of the preferred embodiments of the present invention disclosed above has been provided to enable those skilled in the art to implement and practice the present invention. While the above description has been made with reference to preferred embodiments of the present invention, those skilled in the art will appreciate that various modifications and variations of the present invention, as defined by the following claims, are possible. Accordingly, the present invention is not intended to be limited to the embodiments disclosed herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. In a device for estimating an object position, the device: An image sensor for acquiring images of the road surroundings; and Including a processor that performs object detection from the acquired image, The above processor: Determine whether a part of the first bounding box of the detected object is located in a preset area within the acquired image, or whether a part of the first bounding box is outside the frame boundary of the acquired image; An object position estimation device configured to update the first bounding box to a second bounding box when a portion of the first bounding box is located in a preset area within the acquired image, or when a portion of the first bounding box deviates from the acquired image.

2. In paragraph 1, An object position estimation device, wherein the preset area includes an area in which the detected object is obscured by another fixed object in the acquired image, and thus at least a portion of the detected object is not detected.

3. In paragraph 1, the processor: An object position estimation device configured to generate the second bounding box based on information about the width and height of the bounding box of the object obtained in advance and information about the first bounding box.

4. In the third paragraph, the generation of the second bounding box is An object position estimation device comprising obtaining coordinates of the second bounding box using the width and height of the bounding box of the object obtained in advance, based on a reference edge of the first bounding box that is not located in the preset area.

5. In paragraph 4, The above reference edge is located diagonally with one of the edges of the first bounding box located in the preset area, An object position estimation device, wherein the edge of the selected first bounding box includes an edge whose line connecting a neighboring edge that is not located in the preset area among the edges of the first bounding box located in the preset area has the longest length and belongs to the preset area.

6. In the first paragraph, the generation of the second bounding box is An object position estimation device comprising obtaining coordinates of the second bounding box using the width and height of the bounding box of the object obtained in advance, based on the reference edge of the first bounding box located within the frame boundary of the obtained image.

7. In the third paragraph, an object position estimation device, wherein the center point of the second bounding box is different from the center point of the first bounding box, or at least one of the four edges of the second bounding box is different from the four edges of the first bounding box.

8. In paragraph 1, The processor is configured to determine a current state value of the object by combining the position observation value of the object based on information about the second bounding box and the most recent position prediction state value of the object, An object position estimation device in which the current state value of the above object is used to determine the next position prediction state value.

9. In paragraph 1, An object position estimation device, wherein the processor determines the current state value of the object based on information about the center point coordinates and width and height of the second bounding box.

10. In the first paragraph, a transmitter and receiver for transmitting a traffic safety-related message are included, An object position estimation device, wherein the processor is configured to transmit the traffic safety-related message to a user or user terminal located within a preset distance from a location corresponding to the second bounding box or a location corresponding to a location prediction status value based on the second bounding box through the transceiver.

11. In the first paragraph, a transmitter and receiver for transmitting information about the object are included, An object position estimation device, wherein the processor is configured to transmit information related to a position corresponding to the second bounding box or a position corresponding to a position prediction status value based on the second bounding box to a server via the transceiver.

12. As a method for estimating object location, A step of performing object detection in an image around the acquired road; A step of determining whether a part of the first bounding box of the detected object is located in a preset area within the acquired image or whether a part of the first bounding box is outside the frame boundary of the acquired image; and An object position estimation method, comprising the step of updating the first bounding box to a second bounding box when a part of the first bounding box is located in a preset area within the acquired image or when a part of the first bounding box deviates from the acquired image.

13. In paragraph 12, A method for estimating an object position, comprising a step of generating the second bounding box based on information about the width and height of the bounding box of the object obtained in advance and information about the first bounding box.

14. A method for estimating an object location, comprising the step of transmitting the traffic safety-related message to a user or user terminal within a preset distance from a location corresponding to the second bounding box or a location corresponding to a location prediction status value based on the second bounding box, in the 12th paragraph.

15. A method for estimating an object location, comprising the step of transmitting information related to a location corresponding to the second bounding box or a location corresponding to a location prediction status value based on the second bounding box to a server in the 12th paragraph.

Images