Vehicle position determination method, device and equipment

Abstract

A processor-implemented method includes selecting, in a first frame image corresponding to a first point in time, a first reference landmark from among a first plurality of candidate landmarks associated with a parking area of a vehicle; determining positioning information of the vehicle by using geometric relationship information of the selected first reference landmark with respect to the vehicle; selecting a second reference landmark from among a second plurality of candidate landmarks in a second frame image corresponding to a second time point, the second time point being later in time than the first time point; and updating the positioning information using the geometric relationship information of the selected second reference landmark with respect to the vehicle.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to Korean Patent Application No. 10-2024-0055887, filed on April 26, 2024, with the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference for all purposes. Technical Field

[0003] The following description relates to methods, apparatus, and equipment for determining the location of vehicles. Background Technology

[0004] Semantic road information refers to lanes, pedestrian crossings, stop lines, signs, etc., that provide meaningful information to the driver. In situations with many dynamic objects, stable autonomous driving can be achieved by estimating the position and route of moving objects based on semantic road information. Furthermore, semantic road information can also be utilized when parking in underground parking lots where there may be a large number of moving objects (such as people and vehicles). Summary of the Invention

[0005] This summary is provided to introduce, in a simplified form, some concepts that will be further described in the detailed embodiments described below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to help determine the scope of the claimed subject matter.

[0006] In a general sense, a processor-implemented method is provided, comprising: selecting a first reference landmark from a first plurality of candidate landmarks relating to a parking area of ​​a vehicle in a first frame image corresponding to a first time point; determining vehicle positioning information by using geometrical information of the selected first reference landmark relative to the vehicle; selecting a second reference landmark from a second plurality of candidate landmarks in a second frame image corresponding to a second time point, the second time point being later than the first time point; and updating the positioning information by using geometrical information of the selected second reference landmark relative to the vehicle.

[0007] Determining the vehicle's location information may include determining the vehicle's parking area, and each of the first frame image and the second frame image may include an area corresponding to a portion of the determined parking area, and selecting a first reference landmark may include: detecting a first plurality of candidate landmarks based on the determined parking area; and calculating a score for the detected first plurality of candidate landmarks.

[0008] Determining a parking area can include: identifying the parking area from the area around the vehicle that can be occupied by the vehicle, based on the occupancy information of the space around the vehicle and the vehicle's volume information.

[0009] Detecting the first plurality of candidate landmarks may include detecting the vertices of the parking area as candidate landmarks, and calculating the score of the first plurality of candidate landmarks may include calculating the score of the vertex based on the distance difference between the reference distance between reference vertices at the reference time point and the distance between the first vertices at the first time point.

[0010] Detecting the first plurality of candidate landmarks may include detecting the edges of the parking area as candidate landmarks, and calculating the score of the first plurality of candidate landmarks may include calculating the score of the edge based on the angle of the first edge in the first frame image.

[0011] Detecting the first plurality of candidate landmarks may include detecting objects in or around the parking area as candidate landmarks, and calculating the score of the first plurality of candidate landmarks may include calculating the score of the object based on the angle between the vehicle and the parking area in the first frame image.

[0012] Selecting a first reference landmark may include selecting a first vertex of the parking area in the first frame image as the first reference landmark, and selecting a second reference landmark may include selecting a second vertex of the parking area in the second frame image and a second edge of the parking area in the second frame image as the second reference landmark.

[0013] Updating the vehicle's location information may include: excluding vertices of the parking area in the third frame image from the third reference landmarks based on acquiring a third frame image corresponding to a third time point that is later than the second time point.

[0014] Selecting a first reference landmark may include selecting a first edge of the parking area in the first frame image as the first reference landmark, and selecting a second reference landmark may include selecting a second edge of the parking area in the second frame image and objects in or around the parking area in the second frame image as the second reference landmark.

[0015] Updating the vehicle's location information may include: excluding the third edge of the parking area in the third frame image from the third reference landmark based on acquiring a third frame image corresponding to a third time point that is later than the second time point.

[0016] Updating vehicle location information using geometric relationship information from a second reference landmark may include updating vehicle location information based on a comparison of geometric relationship information from the second reference landmark and the differences between the geometric relationship information.

[0017] Updating vehicle positioning information using geometric relationship information from a second reference landmark may include: transforming the determined positioning information by using inertial data acquired from inertial sensors to obtain comparative geometric relationship information.

[0018] Updating vehicle location information using geometric relationship information from a second reference landmark may include obtaining comparative geometric relationship information by using a map indicating the locations of multiple candidate landmarks.

[0019] Selecting a second reference landmark may include: determining whether to select a candidate landmark as a second reference landmark based on the result of comparing the score of each candidate landmark with a threshold score set for that candidate landmark, and updating the vehicle's positioning information using the geometric relationship information of the second reference landmark may include: skipping the update of the positioning information based on the second frame image in response to the fact that not all of the second plurality of candidate landmarks are selected as the second reference landmark.

[0020] In a general aspect, a processor-implemented method is provided herein, comprising: determining positioning information of a vehicle using first sensing data acquired from a first sensor; determining first geometrical relationship information of landmarks from the first sensing data and second geometrical relationship information of landmarks from the second sensing data based on the scores of landmarks in second sensing data acquired from a second sensor; and adjusting the positioning information by using the difference between the determined first geometrical relationship information and the second geometrical relationship information.

[0021] The first sensor may include an inertial sensor, the first sensing data may include changes in the vehicle's positioning information, and determining the vehicle's positioning information by using the first sensing data may include: in response to acquiring the first sensing data, changing the vehicle's positioning information by using the changes to update the positioning information.

[0022] The second sensor may include a vision sensor, and determining the first geometric relationship information and the second geometric relationship information may include: acquiring an image based on the second sensing data; detecting landmarks from the image; calculating a score for the landmarks detected from the image; and determining the second geometric relationship information based on the score.

[0023] Adjusting location information may include: adjusting location information in response to a score less than a threshold score; and skipping adjustments to location information based on the difference between first geometric relationship information and second geometric relationship information in response to a score greater than or equal to a threshold score.

[0024] The landmark may include multiple candidate landmarks. Determining the first geometric relationship information and the second geometric relationship information may include: calculating the scores of the multiple candidate landmarks in the second sensing data; and determining a reference landmark among the multiple candidate landmarks based on the scores of the multiple candidate landmarks. Adjusting the positioning information includes: adjusting the positioning information based on the difference between the first geometric relationship information and the second geometric relationship information corresponding to each reference landmark.

[0025] Determining the vehicle's location information may include determining the vehicle's parking area, and determining the first geometric relationship information and the second geometric relationship information may include detecting landmarks based on the determined parking area.

[0026] Determining a parking area can include: identifying the parking area within the space that can be occupied by a vehicle, based on space occupancy information and vehicle volume information.

[0027] Detecting landmarks may include detecting vertices of parking areas as landmarks, and determining first geometric relationship information and second geometric relationship information may include: calculating a score for a vertex based on the distance difference between the distance between vertices in the reference geometric relationship information acquired at a reference time point and the distance between vertices in the second sensing data; determining the position of each vertex in the first sensing data as first geometric relationship information in response to the vertex score being less than a vertex threshold score; and determining the position of each vertex in the second sensing data as second geometric relationship information in response to the vertex score being less than a vertex threshold score.

[0028] Detecting landmarks may include detecting the edges of parking areas as landmarks, and determining first geometric relationship information and second geometric relationship information may include: calculating a score of the edge based on the angle of the edge in the second sensing data; determining the relative position of the edge and the vehicle and the angle of the edge and the vehicle in the first sensing data as first geometric relationship information in response to the edge score being less than an edge threshold score; and determining the relative position of the edge and the vehicle and the angle of the edge and the vehicle in the second sensing data as second geometric relationship information in response to the edge score being less than an edge threshold score.

[0029] Detecting landmarks may include detecting objects in or around a parking area as landmarks, and determining first geometric relationship information and second geometric relationship information may include: calculating a score for the object based on the angle between the vehicle and the parking area in the second sensing data; determining the distance from the vehicle to the object in the first sensing data as first geometric relationship information in response to the object's score being less than an object threshold score; and determining the distance from the vehicle to the object in the second sensing data as second geometric relationship information in response to the object's score being less than an object threshold score.

[0030] Determining the first geometric relationship information and the second geometric relationship information may include obtaining the first geometric relationship information by using a map that indicates the location of landmarks.

[0031] In a general sense, a non-transitory computer-readable storage medium is provided herein that stores instructions that, when executed by a processor, cause the processor to perform the method.

[0032] In a general sense, an electronic device is provided herein, comprising a processor configured to execute instructions and a memory storing the instructions, the execution of which configures the processor to: select a first reference landmark from a plurality of candidate landmarks relating to a parking area of ​​a vehicle in a first frame image corresponding to a first time point; determine vehicle positioning information by using geometrical relationship information between the selected first reference landmark and the vehicle; select a second reference landmark from a plurality of candidate landmarks in a second frame image corresponding to a second time point later than the first time point; and update the vehicle positioning information by using geometrical relationship information between the selected second reference landmark and the vehicle.

[0033] In a general aspect, an electronic device is provided herein, comprising a processor configured to execute instructions and a memory storing the instructions, the execution of which configures the processor to: determine positioning information of a vehicle using first sensing data acquired from a first sensor; determine first geometrical relationship information of landmarks from the first sensing data and second geometrical relationship information of landmarks from the second sensing data based on the scores of landmarks in second sensing data acquired from a second sensor; and adjust the positioning information by using the difference between the determined first geometrical relationship information and the second geometrical relationship information.

[0034] Other features and aspects will become clear from the following detailed description, drawings and claims. Attached Figure Description

[0035] Figure 1 An example method for determining the location information of a vehicle according to one or more embodiments is shown.

[0036] Figure 2 Example operations for detecting candidate landmarks and determining a score for a candidate landmark are shown according to one or more embodiments.

[0037] Figure 3 An example of the operation of determining a parking area according to one or more embodiments is shown.

[0038] Figure 4 An example of an operation to change a reference landmark between multiple points in time, according to one or more embodiments, is shown.

[0039] Figure 5 An example of an operation to change a reference landmark between multiple points in time, according to one or more embodiments, is shown.

[0040] Figure 6 An example method is shown according to one or more embodiments for determining the location information of a vehicle by using sensing data collected by multiple sensors.

[0041] Figure 7 Example operations are shown for determining the first geometric relationship information and the second geometric relationship information of the vertices of a parking area according to one or more embodiments.

[0042] Figure 8 Example operations are shown for determining first and second geometric relationship information of the edge of a parking area according to one or more embodiments.

[0043] Figure 9 Example operations are shown for determining first and second geometrical relation information of objects in or around a parking area according to one or more embodiments.

[0044] Figure 10 An example electronic device according to one or more embodiments is shown.

[0045] Throughout the accompanying drawings and detailed description, unless otherwise described or specified, the same reference numerals may be understood to refer to the same or similar elements, features, and structures. The drawings may not be drawn to scale, and for clarity, illustration, and convenience, the relative dimensions, scale, and depiction of elements in the drawings may be enlarged. Detailed Implementation

[0046] The following detailed description is provided to assist the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, after understanding the disclosure of this application, it will become clear that various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will be apparent. For example, the sequence of operations within operations and / or the sequence of operations described herein are merely examples and are not limited to those set forth herein, but will become clear after understanding the disclosure of this application that they can be changed except for the sequence of operations within operations and / or the sequence of operations that must occur in a certain order. As another example, the sequence of operations and / or the sequence of operations within operations can be executed in parallel, except for at least a portion of the sequence of operations and / or at least a portion of the sequence of operations within operations that must occur in a sequential order (e.g., a certain order). Furthermore, for clarity and conciseness, descriptions of features that become apparent after understanding the disclosure of this application may be omitted.

[0047] The features described herein may be implemented in various forms and should not be construed as limited to the examples described herein. Rather, the examples described herein are provided merely to illustrate some of the many feasible ways of implementing the methods, apparatus, and / or systems described herein, which will become clear upon understanding the disclosure of this application.

[0048] Although terms such as “first,” “second,” and “third,” or “A,” “B,” “(a),” and “(b)” may be used herein to describe various components, assemblies, regions, layers, or parts, these components, assemblies, regions, layers, or parts are not limited by these terms. For example, these terms are not used to define the substance, order, or sequence of the corresponding component, assembly, region, layer, or part, but only to distinguish the corresponding component, assembly, region, layer, or part from other components, assemblies, regions, layers, or parts. Therefore, without departing from the teaching of the examples described herein, the first component, assembly, region, layer, or part mentioned in the examples may also be referred to as the second component, assembly, region, layer, or part.

[0049] The terminology used herein is for the purpose of describing various examples only and is not intended to limit this disclosure. Unless the context clearly indicates otherwise, the articles “a,” “an,” and “the” are also intended to include plural forms. As non-limiting examples, the terms “comprising” or “including,” “containing,” and “having” or “possessing” indicate the presence of the stated features, number, operation, component, element, and / or combination thereof, but do not preclude the presence or addition of one or more other features, number, operation, component, element, and / or combination thereof, or alternatively, the presence of alternative features, number, operation, component, element, and / or combination thereof. Furthermore, while one embodiment may illustrate the use of the terms “comprising” or “including,” “containing,” and “having” or “possessing” to indicate the presence of the stated features, number, operation, component, element, and / or combination thereof, other embodiments may exist in which one or more of the stated features, number, operation, component, element, and / or combination thereof are absent.

[0050] As used herein, the term “and / or” includes any one of the listed items and any combination of any two or more. Phrases such as “at least one of A, B, and C”, “at least one of A, B, or C”, etc., are intended to have a separate meaning, and these phrases also include examples such as: unless the corresponding description and embodiments require that such enumeration (e.g., “at least one of A, B, and C”) be interpreted as having a joint meaning, each of A, B, and / or C may exist one or more (e.g., any combination of each of A, B, and C in one or more quantities).

[0051] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains, based on an understanding of the disclosure of this application. Terms such as those defined in common dictionaries shall be interpreted as having the same meaning as in the context of the relevant technology and the disclosure of this application, and shall not be interpreted as having an ideal or overly formal meaning, unless expressly defined herein. In this document, the use of the term “may” (e.g., regarding what an example or embodiment may include or implement) with respect to an example or embodiment means that there exists at least one example or embodiment that includes or implements such a feature, but not that all examples are limited to this.

[0052] Figure 1 An example method for determining the location information of a vehicle according to one or more embodiments is shown.

[0053] In the example, electronic devices (e.g., Figure 10 The electronic device 1000 can determine (e.g., update) the vehicle's positioning information by using geometric relationship information of landmarks in multiple frame images.

[0054] Vehicle positioning information can include the vehicle's position information and its orientation information. Position information can indicate the vehicle's location (or a reference point). Orientation information can indicate the vehicle's orientation (or a reference axis). In this example, the vehicle's positioning information may not include the vertical component, but may include the longitudinal and lateral components.

[0055] In the example, the vehicle's positioning information can indicate its position and / or orientation in a world coordinate system (e.g., an orthogonal coordinate system) determined based on the vehicle's position and orientation at a reference time point. The world coordinate system can be a planar coordinate system. For example, the origin of the world coordinate system can be set to the vehicle's position at the reference time point. The positive direction of the world coordinate system's x-axis can be set to the direction from the rear of the vehicle to the front of the vehicle at the reference time point (e.g., the vehicle's longitudinal direction). The positive direction of the world coordinate system's y-axis can be set to the direction from the user's (e.g., a driver sitting in the vehicle and facing forward) right to left at the reference time point.

[0056] In the example, the electronic device can set the vehicle's location information at a reference time point and can update the vehicle's location information by using sequentially acquired frame images.

[0057] As described in more detail below, information acquired from frame images obtained using a vision sensor (e.g., geometric information about candidate landmarks) can indicate position and / or orientation according to a vehicle coordinate system. The vehicle coordinate system can be a planar coordinate system. For example, the origin of the vehicle coordinate system can be set to the vehicle's position. The positive direction of the x-axis of the vehicle coordinate system can be set from the rear of the vehicle to the front of the vehicle (e.g., the vehicle's longitudinal direction). The positive direction of the y-axis of the vehicle coordinate system can be set from the right to the left of the user (e.g., a driver sitting in the vehicle and facing forward). Unlike the world coordinate system, which is fixed to the vehicle's position and orientation at a reference time point, the origin and axes of the vehicle coordinate system can change with the vehicle's position and orientation. Coordinate transformation between the vehicle coordinate system and the world coordinate system can be performed based on the vehicle's positioning information.

[0058] Electronic devices can control vehicle motion information based on determined (e.g., updated) vehicle positioning information. For example, vehicle motion information may include vehicle speed, vehicle posture, vehicle steering, and / or vehicle gear position.

[0059] refer to Figure 1 In a non-limiting example, in operation 110, the electronic device (e.g., Figure 10 The electronic device 1000 can select a first reference landmark from multiple candidate landmarks related to the parking area of ​​the vehicle in the first frame image corresponding to the first time point.

[0060] In the example, frame images (e.g., a first frame image, a second frame image, or a third frame image) can be acquired by processing sensing data collected by a vision sensor. The vision sensor may include, for example, one or more of a camera sensor, a radar sensor, a lidar sensor, or an ultrasonic sensor. For example, a camera sensor can generate image data as sensing data by receiving and sensing light reflected from a physical point (e.g., light in the visible light band). A radar sensor can generate radar data by emitting and receiving radar signals. A lidar sensor can generate lidar data by emitting and receiving light. An ultrasonic sensor can generate ultrasonic data by emitting and receiving ultrasonic waves. The vision sensor may be mounted on a vehicle and may have a sensing range covering at least a portion of the area surrounding the vehicle. The sensing range of the sensor may refer to the area where information can be acquired using signals emitted and / or received from the sensor (e.g., light, radar signals, and ultrasonic waves).

[0061] In the example, each frame image (e.g., a first frame image, a second frame image, or a third frame image) can be acquired by processing sensing data obtained from multiple vision sensors. For example, each frame image can include an image of the area around the vehicle viewed from the vehicle's vertical direction (e.g., perpendicular to the ground) by processing (e.g., stitching or combining) local images acquired by multiple camera sensors.

[0062] However, in this example, each frame image is not limited to an image viewed from a specific viewpoint direction (e.g., the vertical direction of the vehicle) (e.g., a BEV image). For example, each frame image may refer to an image acquired by a vision sensor placed along the viewpoint direction corresponding to that frame image. For example, each frame image may include an image captured by a single camera sensor.

[0063] In the example, the electronic device can determine the parking area of ​​the vehicle. Each frame image (e.g., a first frame image, a second frame image, or a third frame image) can include an area corresponding to at least a portion of the parking area.

[0064] For example, electronic devices can detect pavement markings (e.g., parking lines) used to define parking areas from images (e.g., frame images) acquired from sensors (e.g., vision sensors). Electronic devices can determine candidate parking areas based on the detected pavement markings. Electronic devices can then designate a candidate parking area as a parking area based on whether the candidate parking area is occupied.

[0065] However, examples are not limited to electronic devices determining parking areas based on road markings; electronic devices can also determine parking areas based on space occupancy information. See below for reference. Figure 3 A more detailed example of determining parking areas based on space occupancy information is provided.

[0066] Electronic devices can detect multiple candidate landmarks based on a defined parking area. The electronic devices can determine or calculate scores for the detected candidate landmarks. The electronic devices can determine whether to designate a candidate landmark as a reference landmark (e.g., a first reference landmark or a second reference landmark) based on the score of each candidate landmark. For example, if the score of each candidate landmark is greater than or equal to a threshold score, the electronic devices can select that candidate landmark as a reference landmark. If the score of each candidate landmark is less than the threshold score for that candidate landmark, the electronic devices can restrict the selection of that candidate landmark as a reference landmark (e.g., they can choose not to select that candidate landmark as a reference landmark). See below for reference. Figure 2 In more detail, the threshold score can be set differently depending on each candidate landmark or the landmark type of each candidate landmark.

[0067] In the example, a landmark may refer to one or more keypoints detected from the frame image. Candidate landmarks may refer to landmarks associated with the parking area, and may include, for example, at least a portion of the parking area's boundary. Reference landmarks may be landmarks selected from the candidate landmarks, and information from the reference landmarks in the frame image (e.g., geometric relationship information) can be used to update the vehicle's location information.

[0068] In the example, the score of a candidate landmark can indicate the reliability of the geometric information of that candidate landmark in a frame image. For example, in a frame image acquired by processing multiple local images, distortion may occur based on or due to the slope of the road surface where the vehicle is located and / or dynamic objects within the sensing range. The score of the candidate landmark can indicate the degree of distortion and / or reliability of the candidate landmark portion of the frame image.

[0069] For example, electronic devices can detect the vertices of a parking area, the edges of the parking area, and objects in or around the parking area as candidate landmarks, and can calculate a score for each candidate landmark. (See below for reference.) Figure 2 The detection of candidate landmarks and the calculation of their scores are described in more detail.

[0070] In the example, in operation 120, the electronic device (e.g., Figure 10 The electronic device 1000 can determine the vehicle's positioning information by using the geometric relationship information between the first reference landmark and the vehicle.

[0071] The geometric relationship information between the reference landmark (e.g., a first reference landmark, a second reference landmark, or a third reference landmark) and the vehicle can refer to the relative position and relative orientation information of the reference landmark relative to the vehicle. In various embodiments of this disclosure, the geometric relationship information between the reference landmark and the vehicle can also be expressed as the geometric relationship information of the reference landmark itself.

[0072] Relative position information can refer to information (e.g., direction or distance) indicating the relative position of a reference landmark based on a specific location (e.g., the position of a vehicle as viewed from a corresponding viewpoint). Relative pose information can refer to the directional relationship (e.g., slope or angle) between a specific location (e.g., the position of a vehicle as viewed from a corresponding viewpoint) and a direction determined by the reference landmark. Geometric relationship information of the reference landmark detected from a frame image corresponding to that specific viewpoint can be determined based on the position and / or orientation of the vehicle at that specific viewpoint.

[0073] The position of a vehicle can be defined by the location of a reference point. For example, the reference point of a vehicle may include a specific point on the rear axle of the vehicle (e.g., the center point). However, in various embodiments of this disclosure, the reference point of a vehicle is not limited to the example of the rear axle of the vehicle, and the reference point of a vehicle may include other points on the vehicle.

[0074] The orientation of a vehicle can be defined based on its posture. For example, orientation based on vehicle posture can include the vehicle's longitudinal direction (or the direction of travel) and its lateral direction.

[0075] For example, if the reference landmark includes points, the geometric relationship information of the reference landmark can include the relative position of the reference landmark based on the vehicle's location. The following will refer to... Figure 7 Provides a more detailed description of the geometric relationships of the reference landmark when the reference landmark is a point.

[0076] For example, if the reference landmark includes a line (e.g., a straight line or a curve), the geometric information of the reference landmark may include the distance from the vehicle's position to the reference landmark and / or the distance from the vehicle's position to the line determined based on the reference landmark. For example, if the reference landmark includes a line, the geometric information of the reference landmark may include the angle and / or slope between the vehicle's orientation and the orientation based on the reference landmark. The following will refer to... Figure 8 Provides a more detailed description of the geometric relationships of the reference landmarks when the reference landmark is a line.

[0077] For example, if the reference landmark includes an object, the geometric information of the reference landmark can include the distance from the vehicle's position to the reference landmark. The distance from the vehicle's position to the reference landmark can include the distance projected from the vehicle's position to the object along a specific direction. This specific direction can be based on the reference landmark or another reference landmark. For example, if the object's shape is linear, the specific direction can be determined as a direction perpendicular to the object. The following will refer to... Figure 9 Provides a more detailed description of the geometric relationships of the reference landmarks when the reference landmark is an object.

[0078] Electronic devices can determine (e.g., correct or update) the vehicle's positioning information by using geometric information from reference landmarks (e.g., a first reference landmark, a second reference landmark, or a third reference landmark). In the example, the electronic device can determine the vehicle's positioning information by using the geometric information from the reference landmarks to correct the vehicle's positioning information estimated by inertial sensors. The following... Figure 1 In operation 140 and refer to Figure 6 A more detailed description is provided of using geometric information from reference landmarks to determine positioning information.

[0079] In operation 130, in the example, the electronic device (e.g., Figure 10 The electronic device 1000 can select a second reference landmark from multiple candidate landmarks in a second frame image corresponding to a second time point that is later than the first time point.

[0080] An electronic device can determine the vehicle's location information based on a first reference landmark selected from a first frame image, and then acquire a second frame image corresponding to a second time point. Similar to selecting the first reference landmark from the first frame image, the electronic device can select a second reference landmark from the second frame image. For example, the electronic device can determine (e.g., detect) at least a portion of a parking area based on the second frame image. The electronic device can detect multiple candidate landmarks based on the parking area. The electronic device can determine scores for the multiple candidate landmarks. The electronic device can determine a second reference landmark for the second frame image based on the scores of the multiple candidate landmarks.

[0081] In the example, the electronic device can select a landmark that has changed from the first reference landmark as the second reference landmark by selecting a first reference landmark and a second reference landmark differently.

[0082] In the example, the electronic device can independently detect candidate landmarks from multiple frames. Due to variations in the field of view (FOV) of the frames, the electronic device may not be able to detect candidate landmarks that were detected in one frame from another frame. In other words, a candidate landmark detected in one frame may be different from a candidate landmark detected in another frame. Therefore, a reference landmark selected from one frame may be different from a reference landmark selected from another frame.

[0083] In the example, even if the electronic device detects at least one candidate landmark from multiple candidate landmarks in multiple frame images, it can independently determine the score of that at least one candidate landmark in each of the multiple frame images. For example, the electronic device can determine the score of a specific candidate landmark in one frame image as a first score, and can determine the score of a specific candidate landmark in another frame image as a second score, which is different from the first score. Even if the same candidate landmark is detected from each of the multiple frame images, the electronic device can determine the score of that candidate landmark differently; therefore, the reference landmark selected from one frame image can be different from the reference landmark selected from another frame image.

[0084] The following will refer to Figure 4 and Figure 5 A more detailed description of an example of selecting reference landmarks from a frame image.

[0085] In the example, if not all candidate landmarks are selected as reference landmarks, the electronic device can skip determining (e.g., updating) the location information. For instance, the electronic device could, for a second frame image, determine whether to select a candidate landmark as a second reference landmark based on the result of comparing the score of each candidate landmark with a threshold score set for that candidate landmark. If not all multiple candidate landmarks in the second frame image are selected as second reference landmarks, the electronic device can skip updating the location information based on the second frame image.

[0086] In the example, in operation 140, the electronic device (e.g., Figure 10 The electronic device 1000 can update the vehicle's positioning information by using the geometric relationship information of the selected second reference landmark from the second frame image of the vehicle.

[0087] The electronic device can determine the geometric relationship information of the second reference landmark. As described above in operation 120, in this example, the electronic device can update the vehicle's location information by using the geometric relationship information of the second reference landmark. The electronic device can update the location information (e.g., the vehicle's location information determined in operation 120) based on the second time point by using the geometric relationship information of the second reference landmark corresponding to the second time point, which reflects the geometric relationship information of the first reference landmark corresponding to the first time point.

[0088] In the example, the electronic device can determine (e.g., update) the vehicle's positioning information based on comparative geometric relationship information and the differences between such information and reference landmarks (e.g., a first or second reference landmark). Comparative geometric relationship information can refer to the geometric relationship information of reference landmarks determined based on information acquired by a visual sensor and another sensor (e.g., an inertial sensor). The geometric relationship information of the reference landmarks (e.g., a first or second reference landmark) can refer to the geometric relationship information of reference landmarks based on information acquired by a visual sensor.

[0089] In other words, the geometric relationship information of the reference landmark and the comparative geometric relationship information can be determined based on data acquired by sensors other than the visual sensor (in some examples, referred to as another sensor, such as an inertial sensor). For example, the geometric relationship information of the reference landmark can be determined based on information acquired by the visual sensor and can be independent of (e.g., regardless of) information acquired by the inertial sensor (e.g., a sensor other than the visual sensor). The comparative geometric relationship information can be determined based on information acquired from both the visual sensor and the inertial sensor (e.g., a sensor other than the visual sensor). Electronic devices can correct the vehicle's positioning information by using the difference between the geometric relationship information of the reference landmark determined by the visual sensor and the comparative geometric relationship information of the reference landmark determined by the inertial sensor.

[0090] For example, comparative geometric information can be acquired based on inertial data. Electronic devices can acquire comparative geometric information by transforming determined positioning information using inertial data acquired from inertial sensors. For example, electronic devices can determine positioning information using the geometric information of a first reference landmark in a first frame image. When acquiring inertial data after determining positioning information, electronic devices can acquire comparative geometric information by transforming the determined positioning information based on the inertial data. Inertial data can indicate changes in vehicle position and / or attitude acquired by inertial sensors. References will follow. Figure 6 A more detailed description is provided of the operation of obtaining comparative geometric relationship information by using inertial data.

[0091] In the example, the electronic device can acquire comparative geometric information by using a map (e.g., a high-density (HD) map) indicating the locations of multiple candidate landmarks. The electronic device can also acquire comparative geometric information by transforming reference geometric information based on inertial data. Reference geometric information can refer to the geometric relationship between a landmark (e.g., a candidate landmark or a reference landmark) and the vehicle at a reference time point (e.g., the parking start time). For example, the electronic device can determine the reference time point based on the determination of the parking area and / or user input. The electronic device can acquire the vehicle's location information corresponding to the reference time point. The electronic device can acquire the vehicle's location information at the reference time point by using a compass and / or a Global Navigation Satellite System (GNSS) device (e.g., a Global Positioning System (GPS) device) installed on the vehicle. The electronic device can acquire reference geometric information for each candidate landmark based on the map and the vehicle's location information corresponding to the reference time point.

[0092] In the example, the map may be stored in the internal memory of the electronic device and / or in external memory accessible by the electronic device. For example, the map may include information (e.g., location or orientation) about candidate landmarks detected in the past (e.g., before the electronic device determines the parking area).

[0093] However, while this description primarily focuses on various examples of electronic devices that determine reference geometric relationship information based on map data, the examples are not limited to this. For instance, the electronic device could acquire geometric relationship information of candidate landmarks detected from frame images corresponding to a reference time point (e.g., a reference frame image) as reference geometric relationship information.

[0094] Figure 2 An example is shown of detecting candidate landmarks and determining a score for the candidate landmark according to one or more embodiments.

[0095] refer to Figure 2 In a non-limiting example, electronic devices (e.g., Figure 10 An electronic device 1000 can determine the parking area 220 of vehicle 210. In this example, vehicle 210 may include electronic devices (e.g., Figure 10 Electronic devices (1000). Figure 2 In the diagram, the x-axis can be the longitudinal direction of the vehicle 210, and the y-axis can be the lateral direction of the vehicle 210. Figure 2 The image shows the vehicle 210 as viewed from the ground in the vertical direction and the area surrounding the vehicle 210. For example, electronic devices can define a square (e.g., a rectangle) on a plane parallel to a plane defined by the x-axis and y-axis as the parking area 220.

[0096] In the example, based on the determined parking area 220, the electronic device can detect multiple candidate landmarks and determine a score for each candidate landmark.

[0097] In the example, the electronic device can detect vertices of parking area 220 (e.g., vertex 221 or vertex 222) as candidate landmarks. The electronic device can also detect a pair of vertices corresponding to the two ends of a local boundary (e.g., an edge or border) as candidate landmarks.

[0098] In the example, the electronic device can extract (e.g., segment) information about the vertices of parking area 220 by applying a frame image to a vertex segmentation model. A vertex segmentation model can refer to a model that is generated and / or trained to detect vertices by applying it to a frame image. The vertex segmentation model can be implemented based on machine learning models and / or neural networks (e.g., convolutional neural networks).

[0099] For example, the electronic device can detect the vertex 221 corresponding to the parking entrance point of the parking area 220 as a candidate landmark. The vertex 221 corresponding to the parking entrance point can refer to the two vertices 221 of the parking area 220 that are closest to the vehicle 210. The vertex 221 corresponding to the parking entrance point can refer to the vertex 221 of the edge that intersects with the vehicle 210 when the vehicle 210 enters the parking area 220.

[0100] However, the examples are not limited to the instance where the electronic device detects vertex 221 corresponding to the parking entrance point as a candidate landmark. The electronic device can detect other vertices as candidate landmarks. For example, the electronic device can detect a pair of vertices 222 among the vertices of parking area 220 that are farthest from vehicle 210 as candidate landmarks. The electronic device can detect a pair of vertices 222 along an edge (parallel to another edge that intersects with vehicle 210 when vehicle 210 enters parking area 220) as candidate landmarks.

[0101] Electronic devices can determine the score of a vertex (or pair of vertices). For example, the score of a vertex can indicate the consistency of the distance between vertices at multiple time points. Electronic devices can determine the score of a vertex based on the distance difference between a reference distance between vertices at a reference time point and the distance between vertices at a specific time point (e.g., a first time point, a second time point, or a third time point).

[0102] An electronic device can obtain the distance between vertices at a reference time point as a reference distance. The electronic device can obtain the reference distance by detecting vertices in a reference frame image and determining the distances between them. However, the example is not limited to this; the electronic device can obtain the distance between vertices determined based on a map, including vertex position information, as the reference distance.

[0103] For example, the electronic device can detect vertices as candidate landmarks from a reference frame image corresponding to a reference time point (e.g., time point 0). The electronic device can determine the coordinates of the vertices of vehicle 210 in the world coordinate system at the reference time point based on the reference frame image.

[0104] Equation 1:

[0105]

[0106] In the non-restrictive example, in Equation 1, Represents the coordinates of the first vertex (e.g., the left vertex) of the reference frame image. Represents the coordinates of the second vertex (e.g., the right vertex) of the reference frame image. Represents the first vertex x-coordinate, Represents the first vertex y-coordinate, Indicates the second vertex The x-coordinate, and Indicates the second vertex The y-coordinate. For reference, the coordinate system of vehicle 210 at the reference time point can be the same as the world coordinate system.

[0107] For example, the electronic device can detect vertices as candidate landmarks from the k-th frame image corresponding to time point k. The electronic device can determine the coordinates of the vertices according to the coordinate system of vehicle 210 based on the k-th frame image.

[0108] Equation 2:

[0109]

[0110] In the non-restrictive example, in Equation 2, This represents the coordinates of the first vertex (e.g., the left vertex) of the k-th frame image. This represents the coordinates of the second vertex (e.g., the right vertex) of the k-th frame image. Represents the first vertex x-coordinate, Represents the first vertex y-coordinate, Indicates the second vertex The x-coordinate, and Indicates the second vertex The y-coordinate.

[0111] Electronic devices can determine the score of a vertex based on the distance difference between vertices.

[0112] Equation 3:

[0113]

[0114] In the non-restrictive example, in Equation 3, This represents the score of the vertices in the k-th frame of the image. This represents the distance between vertices in the k-th frame of the image. This represents the reference distance between vertices in the reference frame image, and This represents the distance difference between the reference distance and the distance between vertices in the k-th frame image.

[0115] Electronic devices can be based on the fraction of vertices. With vertex threshold score thr p The result of the comparison determines whether to select a vertex as a reference landmark. For example, when the vertex's score... Less than the vertex threshold score thr pAt that time, the electronic device can select a vertex as a reference landmark. When the vertex's score... Greater than or equal to the vertex threshold score thr p At that time, electronic devices may not select vertices as reference landmarks.

[0116] In the example, the electronic device can detect the edges of parking area 220 as candidate landmarks. The edges of parking area 220 can refer to at least a portion of the boundary of parking area 220. The boundary can be divided into multiple local boundaries based on the change in the slope (e.g., curvature) of the tangent at each point of the boundary of parking area 220. Edges can be defined by the local boundaries of parking area 220. For example, when parking area 220 is a polygon, the local boundaries or edges of parking area 220 can be defined by edges. However, although the actual parking area 220 has a polygonal shape, at least a portion of the boundary of parking area 220 may be curved due to distortion and / or errors in the frame image.

[0117] In the example, the electronic device can detect parallel edge pairs (e.g., edges 231 and 232, and edges 233 and 234) as candidate landmarks from the edges of the parking area 220. For example, the electronic device can detect parallel edges (e.g., edges 231 and 232, and edges 233 and 234) as candidate landmarks from the edges of the parking area 220. When the vehicle 210 is parked in the parking area 220, the electronic device can detect edges parallel to the longitudinal direction of the vehicle 210 (e.g., edges 231 and 232) as candidate landmarks. For example, when the vehicle 210 is parked in the parking area 220, the electronic device can detect edges intersecting the longitudinal direction of the vehicle 210 (e.g., vertical edges) (e.g., edges 233 and 234) as candidate landmarks.

[0118] However, the examples are not limited to the example of the electronic device detecting parallel edge pairs as candidate landmarks, and the electronic device can detect a pair of edges (e.g., edges 231 and 234) adjacent to the vertex of parking area 220 as candidate landmarks from among the edges of parking area 220.

[0119] In the example, the electronic device can extract (e.g., segment) information about the edges of parking area 220 by applying a frame image to an edge segmentation model. An edge segmentation model can refer to a model generated and / or trained to detect edges or regions corresponding to edges by applying it to a frame image. The edge segmentation model can be implemented based on machine learning models and / or neural networks (e.g., convolutional neural networks).

[0120] Electronic devices can determine the score of an edge (or pair of edges). The score of an edge can indicate the angle between the edges. When identifying parallel edges of parking area 220 as candidate landmarks, electronic devices can determine the score of the edges based on the angle of the edges in a frame image.

[0121] In the example, the electronic device can detect parallel edges 231 and 232 of the parking area 220 from the k-th frame image as candidate landmarks. The electronic device can model (e.g., fit or regress) each edge in the parallel edges as a straight line according to the coordinate system of the vehicle 210 corresponding to the k-th time point.

[0122] Equation 4:

[0123]

[0124] In the non-restrictive example, in Equation 4, This represents the straight line corresponding to the first edge 231 (e.g., the left edge) of the k-th frame image, and This represents the straight line corresponding to the second edge 232 (e.g., the right edge) of the k-th frame image.

[0125] In the example, the electronic device can determine the score of an edge based on the slope difference between the edges, which indicates the angle between the edges. The electronic device can determine whether the number of pixels divided into each edge exceeds a threshold number of pixels. When the total number of pixels of each edge in a segmented parallel edge (e.g., edges 231 and 232) exceeds the threshold number of pixels, the electronic device can determine the score of the edge based on the slope difference of the edges. When the number of pixels of at least one edge in a segmented parallel edge (e.g., edges 231 and 232) is less than or equal to the threshold number of pixels, the electronic device can determine that the score is a predetermined value (e.g., a value greater than or equal to the threshold score).

[0126] Equation 5:

[0127]

[0128] In the non-restrictive example, in Equation 5, The fraction representing the edges (e.g., edges 231 and 232) of the parking area 220 in the k-th frame image. This indicates the number of pixels that were divided into the first side 231. This represents the number of pixels that are segmented into the second edge 2^32, and L represents the threshold number of pixels used to determine whether the segmented edge is valid. This represents the slope of the first side 231. ∞ represents the slope of the second side 232, and ∞ represents a predetermined value that is greater than or equal to the threshold fraction.

[0129] In the example, the electronic device could be based on a fraction of the edges. With edge threshold score thr l The results of the comparison determine whether to select the edge as a reference landmark. For example, when the edge's score... Less than the edge threshold score thr l At that time, the electronic device can select the edge as a reference landmark. When the edge score Greater than or equal to the edge threshold score thr l At that time, electronic devices may not select the edge as a reference landmark.

[0130] In the example, the electronic device can detect objects in or around parking area 220 as candidate landmarks. Objects can refer to objects with a vertical height greater than or equal to a threshold height. Objects in parking area 220 may include, for example, parking barriers. Objects around parking area 220 may include pillars or flower beds.

[0131] Electronic devices can segment regions corresponding to or around objects in or around parking area 220 by applying the k-th frame image to an object segmentation model. The object segmentation model can refer to a model generated and / or trained to detect objects (e.g., predetermined objects) by applying it to frame images. The object segmentation model can be implemented based on machine learning models and / or neural networks (e.g., convolutional neural networks).

[0132] Electronic devices can determine the score of an object based on the angle between vehicle 210 and parking area 220 in a frame image (e.g., a first frame image, a second frame image, or a third frame image). In the example, when the object's height in the vertical direction of vehicle 210 is greater than or equal to a threshold height, the distortion of the object in the frame image may increase as the angle between vehicle 210 and parking area 220 increases. Electronic devices can indicate the degree of distortion in a frame image by using the angle of vehicle 210 relative to parking area 220.

[0133] In the example, the electronic device can determine the score of the object based on the average slope of the parallel edges (e.g., edges 231 and 232) within the edge of the parking area 220. When the vehicle 210 is parked in the parking area 220, the parallel edges can include edges parallel to the longitudinal direction of the vehicle 210. When the number of pixels segmented into the object exceeds a threshold number of pixels, the electronic device can determine the score of the object based on the average slope of the parallel edges of the parking area 220. When the number of pixels segmented into the object is less than or equal to the threshold number of pixels, the electronic device can determine that the score of the object is a predetermined value (e.g., a value greater than or equal to the threshold score).

[0134] Equation 6:

[0135]

[0136] In the non-restrictive example, in Equation 6, Let represent the score of the object in the k-th frame image, s represent the number of pixels segmented into the object, and S represent the number of thresholds used to determine whether the segmented object is valid. This represents the slope of the first side 231. ∞ represents the slope of the second side 232, and ∞ represents a predetermined value that is greater than or equal to the threshold fraction.

[0137] In the example, the electronic device can be based on the score of the object. With object threshold score thr s The result of the comparison determines whether to select the object as a reference landmark. For example, when the object's score... Less than the object threshold score thr s At that time, the electronic device can select the object as a reference landmark. When the object's score Greater than or equal to the object threshold score thr s At that time, the electronic device may not select the object as a reference landmark.

[0138] Figure 3 An example of determining a parking area according to one or more embodiments is shown.

[0139] refer to Figure 3 In a non-limiting example, the electronic equipment of vehicle 310 (e.g., Figure 10 The electronic device 1000 can determine the parking area. The electronic device can determine the parking area within the area around the vehicle 310 that can be occupied by the vehicle 310 based on the occupancy information of the space around the vehicle 310 and the volume information of the vehicle 310.

[0140] In the example, the volume information of vehicle 310 may include information about the space occupied by vehicle 310. For example, the volume information of vehicle 310 may include the dimensions of the solid bounding box (e.g., a 3D bounding box) surrounding vehicle 310. The dimensions of the solid bounding box may include the longitudinal and lateral lengths of vehicle 310. The dimensions of the solid bounding box may also include the vertical length.

[0141] In the example, occupancy information of the space surrounding vehicle 310 may include the occupancy status of the space surrounding the vehicle. For example, occupancy information may indicate whether a unit space is occupied or unoccupied for each of the multiple unit spaces (e.g., voxels) into which the space surrounding vehicle 310 is divided. For example, occupancy information may indicate that the status of each unit space is occupied, unoccupied, or unknown.

[0142] In the example, occupancy information can be determined based on data acquired by sensors (e.g., vision sensors) installed on vehicle 310. However, the example is not limited to determining occupancy information based on data acquired by sensors installed on vehicle 310, and occupancy information can also be received from a server that manages the space around the vehicle.

[0143] In the example, such as Figure 3 As shown, another vehicle 320 may be parked there, encroaching on or crossing the parking line defined by road markings. Electronic devices can determine the space around vehicle 310 that can be occupied by vehicle 310, excluding the area occupied by the object (e.g., another vehicle 320). The electronic devices can determine the parking area based on a comparison of the volume information of vehicle 310 with the available space. For example, when an area 330 within the area defined by road markings, excluding the space occupied by another vehicle 320, can cover or be sufficiently spacious to include the volume information of vehicle 310, the electronic devices can determine that area 330 as a parking area. In this case, the local boundaries (e.g., edges) of the parking area can be determined based on the other vehicle 320.

[0144] Figure 4 An example of changing a reference landmark between multiple points in time, according to one or more embodiments, is shown.

[0145] refer to Figure 4 In a non-limiting example, electronic devices (e.g., Figure 10 The electronic device 1000 can update the positioning information of the vehicle 440 by using multiple frame images corresponding to multiple time points, based on the parking status of the vehicle 440. (See above reference...) Figure 1 The reference landmarks selected from each frame image can be different. For example, before the vehicle 440 enters the parking area 400, the electronic device can determine the vertex 401 of the parking area 400 as a reference landmark. When the vehicle 440 enters the parking area 400, the electronic device can further determine the edge 402 of the parking area 400 together with the vertex 401 of the parking area 400 as reference landmarks. When the vehicle 440 further enters the parking area 400, the electronic device can determine only the edge 402 of the parking area 400 as a reference landmark, without determining the vertex 401 of the parking area 400 as a reference landmark.

[0146] The electronic device can acquire the first frame image corresponding to the first time point 410. At the first time point 410, such as... Figure 4 As shown, vehicle 440 may not have entered parking area 400.

[0147] The electronic device can select vertex 401 of parking area 400 in the first frame image as a first reference landmark. For example, the electronic device can detect vertex 401 of parking area 400 in the first frame image. The electronic device can determine a score for vertex 401. The electronic device can determine vertex 401 as the first reference landmark based on determining that the score of vertex 401 is less than a vertex threshold score. The electronic device can detect at least some of the edges 402 of parking area 400 in the first frame image. The electronic device can determine a score for edge 402. The electronic device can exclude (e.g., not select) edge 402 from the first reference landmark based on determining that the score of edge 402 is greater than or equal to an edge threshold score.

[0148] The electronic device can acquire a second frame image corresponding to the second time point 420. At the second time point 420, such as... Figure 4 As shown, vehicle 440 may have begun to enter parking area 400.

[0149] The electronic device can select vertices 401 and edges 402 of parking area 400 in the second frame image as second reference landmarks. For example, the electronic device can also select edges 402 of parking area 400 as reference landmarks. For example, the electronic device can detect vertices 401 of parking area 400 in the second frame image. The electronic device can determine a score for vertex 401. The electronic device can determine vertex 401 as a second reference landmark based on the determination that the score of vertex 401 is less than a vertex threshold score. The electronic device can detect at least some of the edges 402 of parking area 400 in the second frame image. The electronic device can determine the score of edge 402. Compared to the first time point 410, the score of edge 402 may decrease at the second time point 420. The electronic device can select edge 402 as a second reference landmark based on the determination that the score of edge 402 is less than an edge threshold score.

[0150] The electronic device can acquire a third frame image corresponding to the third time point 430. At the third time point 430, such as... Figure 4 As shown, compared to the second time point 420, vehicle 440 can further enter parking area 400.

[0151] The electronic device can exclude vertices 401 of the parking area 400 in the third frame image from the third reference landmarks based on acquiring a third frame image corresponding to a third time point 430 that is later in time than the second time point 420. For example, the electronic device can detect vertices 401 of the parking area 400 in the third frame image. The electronic device can determine a score for vertex 401. Compared to the first time point 410 and / or the second time point 420, the score of vertex 401 may increase at the third time point 430. The electronic device can exclude (e.g., not select) vertex 401 from the third reference landmarks based on determining that the score of vertex 401 is greater than or equal to a vertex threshold score. The electronic device can detect at least some of the edges 402 of the parking area 400 in the third frame image. The electronic device can determine the score of the edges 402. The electronic device can select edge 402 as a third reference landmark based on determining that the score of edge 402 is less than an edge threshold score.

[0152] Figure 5 An example of changing a reference landmark between multiple points in time, according to one or more embodiments, is shown.

[0153] refer to Figure 5 In a non-limiting example, electronic devices (e.g., Figure 10 The electronic device 1000 can update the positioning information of the vehicle 540 by using multiple frame images corresponding to multiple time points, based on the parking status of the vehicle 540. For example, when the vehicle 540 is entering the parking area 500, the electronic device can identify the edge 502 of the parking area 500 as a reference landmark. As the vehicle 540 further enters the parking area 500, the electronic device can further identify objects 503 in the parking area 500 together with the edge 502 of the parking area 500 as reference landmarks. When the parking of the vehicle 540 is at least substantially completed, the electronic device can identify only objects 503 in the parking area 500 as reference landmarks, without identifying the edge 502 of the parking area 500 as reference landmarks.

[0154] In the example, the electronic device can acquire a first frame image corresponding to the first time point 510. At the first time point 510, such as Figure 5 As shown, part of vehicle 540 may have already entered parking area 500. Figure 5 The first time point 510 can basically correspond to Figure 4 The third time point is 430.

[0155] The electronic device can select an edge 502 of a parking area 500 in a first frame image as a first reference landmark. For example, the electronic device can detect an edge 502 of the parking area 500 in the first frame image. The electronic device can determine a score for the edge 502. The electronic device can determine the edge 502 as a first reference landmark based on the determination that the score of the edge 502 is less than an edge threshold score. The electronic device can detect at least some (i.e., a portion) of objects 503 in the parking area 500 in the first frame image. The electronic device can determine a score for the objects 503. The electronic device can exclude (e.g., not select) objects 503 from the first reference landmarks based on the determination that the score of the objects 503 is greater than or equal to an object threshold score.

[0156] In the example, the electronic device can acquire a second frame image corresponding to the second time point 520. At the second time point 520, as... Figure 5 As shown, compared to the first time point 510, vehicle 540 can further enter the parking area 500.

[0157] The electronic device can select the edge 502 of the parking area 500 in the second frame image and objects 503 in or around the parking area 500 in the second frame image as second reference landmarks. For example, the electronic device can also select objects 503 in or around the parking area 500 as reference landmarks. For example, the electronic device can detect the edge 502 of the parking area 500 in the second frame image. The electronic device can determine the score of the edge 502. The electronic device can determine the edge 502 as a second reference landmark based on the determination that the score of the edge 502 is less than an edge threshold score. The electronic device can detect at least some of the objects 503 in the parking area 500 in the second frame image. The electronic device can determine the score of the objects 503. Compared with the first time point 510, the score of the objects 503 may decrease at the second time point 520. The electronic device can select the objects 503 as second reference landmarks based on the determination that the score of the objects 503 is less than an object threshold score.

[0158] In the example, the electronic device can acquire a third frame image corresponding to the third time point 530. At the third time point 530, as... Figure 5 As shown, compared to the second time point 520, vehicle 540 can further enter the parking area 500.

[0159] The electronic device can exclude edges 502 of parking area 500 in the third frame image from the third reference landmark based on acquiring a third frame image corresponding to a third time point 530 that is later in time than the second time point 520. For example, the electronic device can detect edges 502 of parking area 500 in the third frame image. The electronic device can determine a score for edge 502. Compared to the first time point 510 and / or the second time point 520, the score of edge 502 may increase at the third time point 530. The electronic device can exclude (e.g., not select) edge 502 from the third reference landmark based on determining that the score of edge 502 is greater than or equal to an edge threshold score. The electronic device can detect at least some (i.e., a portion) of objects 503 in or around parking area 500 in the third frame image. The electronic device can determine a score for object 503. The electronic device can select object 503 as the third reference landmark based on determining that the score of object 503 is less than an object threshold score.

[0160] In the example, for ease of description, the following are described respectively. Figure 4 and Figure 5 However, the description is not limited to this. For example, Figure 4 The first time point 410 Figure 4 The second time point 420 Figure 4 The third time point 430 Figure 5 The second time point 520 Figure 5 The third time point 530 can have a temporal sequence. For example, Figure 4 The third time point 430 can be compared with Figure 5 The first time point 510 is basically the same as or corresponds to it.

[0161] In summary, depending on the parking progress, the electronic device can use vertex 401 of parking area 400 or 500, edge 402 of parking area 400 or 500, and object 503 in or around parking area 400 or 500 as reference landmarks. For example, at the start of parking, the electronic device can use vertex 401 of parking area 400 or 500 as a reference landmark. Then, depending on the parking progress, the electronic device can also use edge 402 of parking area 400 or 500 along with vertex 401 of parking area 400 or 500 as reference landmarks. Then, depending on the parking progress, the electronic device can stop using vertex 401 of parking area 400 or 500 as a reference landmark. Then, depending on the parking progress, the electronic device can use object 503 in or around parking area 400 or 500 along with edge 402 of parking area 400 or 500 as reference landmarks. Depending on the parking progress, the electronic equipment can stop using the edge 402 of parking area 400 or the edge 502 of parking area 500 as a reference landmark. In the final stage of parking, the electronic equipment can update the location information of vehicle 540 by using objects 503 in or around parking area 400 or 500 as reference landmarks.

[0162] Figure 6 An example is shown of determining the vehicle's location information using sensing data collected by multiple sensors, according to one or more embodiments.

[0163] refer to Figure 6 In a non-limiting example, electronic devices (e.g., Figure 10 Electronic devices 1000 can determine (e.g., update) the vehicle (e.g., by using sensing data acquired from multiple sensors). Figure 4 Vehicle 440 and Figure 5 Location information of vehicle 540.

[0164] In the example, the vehicle's location information x k It can be represented as shown in Equation 7 below.

[0165] Equation 7:

[0166]

[0167] In the non-restrictive example, in Equation 7, x k x represents the location information of the vehicle corresponding to the k-th time point. k The x-coordinate and y-coordinate indicate the vehicle's position at time point k. k Let θ represent the y-coordinate indicating the vehicle's position at time point k, and let θ be the coordinate of the vehicle's position at time point k. kThis represents the vehicle's rotation angle relative to a reference axis (e.g., the x-axis). As described below, the index (e.g., k) of the vehicle's positioning information corresponds to a sampling time point of a specific sensor (e.g., a vision sensor), and this index can be updated when the sensed data (e.g., frame images) has been fully processed. The sampling period of another sensor (e.g., an inertial sensor) may be shorter than that of the specific sensor, and the determination and / or update performed on the positioning information in response to sensed data acquired by the other sensor (e.g., the inertial sensor) may be independent of changes in the index.

[0168] In the example, the electronic device can determine (e.g., initialize) the vehicle's location information at a reference time point, as shown in Equation 8 below.

[0169] Equation 8:

[0170]

[0171] In the example, the vehicle's location information can be represented based on a world coordinate system. (See the reference above.) Figure 1 The origin of the world coordinate system can be set to the vehicle's position at the reference time point. The positive direction of the x-axis of the world coordinate system can be set to the direction from the rear of the vehicle to the front of the vehicle at the reference time point (e.g., the vehicle's longitudinal direction). The positive direction of the y-axis of the world coordinate system can be set to the direction from the right to the left of the user (e.g., the driver sitting in the vehicle and facing forward) at the reference time point.

[0172] Electronic devices can update the vehicle's location information at a reference time point by using sensing data acquired from multiple sensors, and can determine the vehicle's location information.

[0173] In the example, in operation 610, the electronic device (e.g., Figure 10 The electronic device 1000 can determine the vehicle's location information by using first sensing data obtained from the first sensor.

[0174] The first sensor may include an inertial sensor. An inertial sensor can refer to a sensor that measures inertial forces. For example, an inertial sensor may include a sensor that measures the velocity, angular velocity, direction, gravity, acceleration, and / or angular acceleration of a moving object. For example, an inertial sensor may be implemented as at least part of an inertial measurement unit (IMU).

[0175] In the example, the inertial sensor may include an accelerometer and / or a gyroscope sensor. An accelerometer can sense and measure acceleration applied to an object. For example, an accelerometer can measure acceleration relative to three axes (e.g., the x-axis, y-axis, and z-axis). Since an accelerometer senses acceleration in the direction of gravity when at rest, it can also measure acceleration in a direction perpendicular to the ground, similar to gravitational acceleration g. A gyroscope sensor (also known as a gyroscope) can measure angular velocity by converting the Coriolis force generated by rotational motion into an electrical signal.

[0176] The first sensing data may include changes in the vehicle's positioning information. These changes may include variations in time, speed, and angular velocity. The time variation may refer to the length of time from the previous sampling point of the first sensing data to the current sampling point.

[0177] Electronic devices can update vehicle location information in response to acquiring initial sensing data. For example, electronic devices can update the location information by transforming the vehicle's location information using change information.

[0178] For example, the vehicle's location information can be updated based on the first sensing data, as shown in Equation 9 below.

[0179] Equation 9:

[0180]

[0181] In the unrestricted example, in Equation 9, dt represents the change in time, and v k Represents velocity, and ω k It represents angular velocity.

[0182] In the example, in operation 620, the electronic device (e.g., Figure 10 An electronic device 1000 can determine first geometric relationship information of landmarks from the first sensing data and second geometric relationship information of landmarks from the second sensing data based on the scores of landmarks in the second sensing data acquired from the second sensor. For example, the second sensor may include a vision sensor.

[0183] The first geometric relationship information can refer to the geometric relationship information of a landmark determined based on first sensing data acquired from an inertial sensor (also referred to as inertial data in various examples herein). For example, the first geometric relationship information can refer to the geometric relationship information of a landmark determined based on inertial data. For reference, since the first geometric relationship information is the geometric relationship information of a landmark determined based on inertial data, the first geometric relationship information can correspond to the reference above. Figures 1 to 5 The aforementioned comparative geometric relationship information.

[0184] The second geometric relationship information can refer to the geometric relationship information of a landmark determined based on second sensing data acquired from a visual sensor (also referred to as an image or frame image in various embodiments herein). For example, the second geometric relationship information can refer to the geometric relationship information of a landmark determined in an image. For reference, since the second geometric relationship information is the geometric relationship information of a landmark determined based on an image, the second geometric relationship information can correspond to the reference above. Figures 1 to 5 The geometric relationship information of the specific time points (e.g., the first time point, the second time point, and the third time point).

[0185] In the example, the electronic device can acquire an image (e.g., a frame image) based on second sensing data. The electronic device can detect landmarks from the image. The electronic device can determine a score for the landmarks detected from the image. The electronic device can then determine second geometric relationship information based on this score.

[0186] As referenced above Figure 1 The images may include, but are not limited to, images of the area surrounding the vehicle viewed from a vertical direction (e.g., a direction perpendicular to the ground) obtained by processing (e.g., stitching or combining) local images acquired by multiple camera sensors. Each image may refer to an image acquired by a vision sensor placed along a viewpoint direction corresponding to that image.

[0187] As referenced above Figures 1 to 5 As described in the example, the electronic device can determine the parking area of ​​a vehicle. The electronic device can then detect landmarks based on the parking area. This is due to the determination of the parking area and the detection and reference of landmarks. Figures 1 to 5 The elements described are the same, so repeated descriptions are omitted.

[0188] In the example, the electronic device can determine whether to determine first geometric relationship information and second geometric relationship information, and whether to adjust the positioning information, based on the scores of landmarks detected from the second sensing data. For example, if the score of a landmark is less than a threshold score, the electronic device can determine the first geometric relationship information and second geometric relationship information, and can adjust the positioning information. For example, if the score of a landmark is greater than or equal to the threshold score, the electronic device can skip determining the first geometric relationship information and second geometric relationship information and / or adjust the positioning information based on the second geometric relationship information.

[0189] As mentioned above, a landmark can be identified as the apex of a parking area, the edge of a parking area, and / or an object within or around a parking area. References will follow below. Figures 7 to 10 The operation of determining the first and second geometric relationship information for each landmark is described in more detail.

[0190] In the example, in operation 630, the electronic device (e.g., Figure 10 The electronic device 1000 can adjust the positioning information by using the difference between the determined first geometric relationship information and the second geometric relationship information.

[0191] Electronic devices can adjust positioning information by using, for example, the difference between first and second geometric relation information applied through an extended Kalman filter (EKF), which is discussed in more detail below with reference to Equation 10.

[0192] Equation 10:

[0193]

[0194] In the non-restrictive example, in Equation 10, x k+1 Let x represent the location information of the vehicle at time point k+1, and x k This represents the location information of the vehicle at time point k. Represents the first geometric relation information, and z k This represents the second geometric relationship information. K k Indicates the Kalman filter gain. x represents the vehicle's location information before image-based adjustments. k In the error, Q represents the value set for the error in the inertial data, and A... k Let H represent the coefficients used to linearly represent the updates to the vehicle's positioning information based on inertial data, and H... k This represents a value determined based on each landmark. H k See below for reference. Figures 7 to 9 To describe in more detail.

[0195] As referenced above Figures 1 to 5 In the example described above, the electronic device can detect multiple candidate landmarks and determine a reference landmark among them to be used for determining (e.g., adjusting or updating) positioning information. For example, the electronic device can detect multiple candidate landmarks from an image. The electronic device can determine scores for the multiple candidate landmarks in the image. The electronic device can determine a reference landmark among the multiple candidate landmarks based on their scores. The electronic device can adjust the positioning information based on the difference between first and second geometric relationship information corresponding to each reference landmark. Because the detection of candidate landmarks, the determination of their scores, and the selection of reference landmarks are related to the above... Figures 1 to 5 The elements described are the same, so repeated descriptions are omitted.

[0196] Figure 7Examples of first and second geometric relation information for determining the vertices of a parking area according to one or more embodiments are shown.

[0197] refer to Figure 7 In a non-limiting example, electronic devices (e.g.,) may be installed in vehicle 700. Figure 10 The electronic device 1000 can detect vertices 711 and 712 of the parking area as landmarks (or candidate landmarks). The electronic device can determine the scores of vertices 711 and 712. (See above reference.) Figure 2 The electronic device can determine the scores of vertices 711 and 712 based on the distance difference between a reference distance and the distance between vertices 711 and 712 in the image frame. The electronic device can also determine first geometric relationship information and second geometric relationship information for vertices 711 and 712 based on the fact that the scores of vertices 711 and 712 are less than a vertex threshold score.

[0198] The electronic device can determine the position of each vertex in the first sensing data as first geometric relationship information. The position of each vertex in the first sensing data can be represented by coordinates according to the vehicle coordinate system. For example, when the image corresponds to the k-th time point, the first geometric relationship information of vertices 711 and 712 according to the vehicle coordinate system at the k-th time point can be represented as shown in Equation 11 below.

[0199] Equation 11:

[0200]

[0201] In the non-restrictive example, in Equation 11, similar to Equation 1, This represents the x-coordinate of the first vertex 711 (e.g., the left vertex) in the world coordinate system. This represents the y-coordinate of the first vertex 711 in the world coordinate system. This represents the x-coordinate of the second vertex 712 (e.g., the right vertex) according to the world coordinate system, and This represents the y-coordinate of the second vertex 712 in the world coordinate system.

[0202] Similar to Equation 7, in the world coordinate system, as shown in Equation 11 above, x k The x-coordinate and y-coordinate indicate the position of vehicle 700 at time point k. k Let θ represent the y-coordinate indicating the position of vehicle 700 at time point k, and let θ be the coordinate of the vehicle's position. k This indicates the rotation angle of vehicle 700 relative to a reference axis (e.g., the x-axis).

[0203] In equation 11, The information represents the first geometric relationship between vertices 711 and 712, and may include the coordinates of the first vertex 711 and the second vertex 712 to which a coordinate system transformation from the world coordinate system to the vehicle coordinate system at the k-th time point has been performed. To transform from the world coordinate system to the vehicle coordinate system, the electronic device may base its transformation on the position information of the vehicle 700 at the k-th time point (e.g., x on the x-axis). k y on the y-axis k Translation is performed on the coordinates of the first vertex 711 and the second vertex 712 according to the world coordinate system, and then the translation can be based on the pose information of the vehicle 700 (e.g., -θ). k To perform the rotation.

[0204] In the example, the electronic device can determine the position of each vertex in the second sensing data as second geometric relation information. Similar to the position of each vertex in the first sensing data, the position of each vertex in the second sensing data can be represented by coordinates according to the vehicle coordinate system. For example, when the image corresponds to the k-th time point, the second geometric relation information of vertices 711 and 712 according to the vehicle coordinate system at the k-th time point can be represented as shown in Equation 12 below.

[0205] Equation 12:

[0206]

[0207] In the non-restrictive example, in Equation 12, This represents the second geometric relationship information between vertices 711 and 712. and Let x and y represent the x and y coordinates of the first vertex 711 (e.g., the left vertex) in the vehicle coordinate system at time point k, respectively. and These represent the x and y coordinates of the second vertex 712 (e.g., the right vertex) in the vehicle coordinate system at time point k, respectively.

[0208] Electronic devices can adjust the vehicle's positioning information based on the difference between the first and second geometric relationship information of vertices 711 and 712.

[0209] In a non-limiting example, the electronic device may adjust the vehicle's positioning information based on Equation 13 as shown below.

[0210] Equation 13:

[0211]

[0212] Figure 8 Examples of first and second geometric relationship information for determining the edges of a parking area according to one or more embodiments are shown.

[0213] In the example, the electronic device can detect the edges of the parking area as landmarks (or candidate landmarks). The electronic device can then determine a score for each edge. (See the reference above.) Figure 2 The electronic device can detect the edge score based on the edge angle. The electronic device can determine first geometric relationship information and second geometric relationship information of the edge based on the edge score being less than an edge threshold score.

[0214] The electronic device can determine first geometric relationship information from the relative position of the edge to the vehicle and the angle between the vehicle and the edge in the first sensing data. The electronic device can determine second geometric relationship information from the relative position of the edge to the vehicle and the angle between the vehicle and the edge in the second sensing data.

[0215] The relative position of the edge to the vehicle can include the distance from the centerline of the edge to the vehicle's position. For example, the relative position can include the length obtained by projecting the distance from the centerline of the edge to the vehicle's position onto a local boundary (e.g., one side) of the parking area.

[0216] The angle between the vehicle and the edge can include the angle between the vehicle's direction of travel and the edge's representative direction. The edge's representative direction can be determined based on the edge's corresponding direction.

[0217] See Figure 8 In a non-restrictive example, the first geometric relationship information of the edge can be determined based on Equation 14 shown below.

[0218] Equation 14:

[0219]

[0220] In the non-restrictive example, in Equation 14, The first geometric relationship information representing the edge of the parking area, and It represents the length obtained by projecting the distance from the centerline of the edge to the vehicle's position onto the local boundary (e.g., one side) of the parking area, as well as the positional relationship between the edge and the vehicle.

[0221] Similar to Equation 1, in Equation 14, Represents the coordinates of the first vertex (e.g., the left vertex) of the reference frame image. Represents the coordinates of the second vertex (e.g., the right vertex) of the reference frame image. This represents the x-coordinate of the first vertex (e.g., the left vertex) in the world coordinate system. This represents the y-coordinate of the first vertex in the world coordinate system. This represents the x-coordinate of the second vertex (e.g., the right vertex) in the world coordinate system. This represents the y-coordinate of the second vertex in the world coordinate system.

[0222] Similar to Equation 7, in Equation 14, x is a value conforming to the world coordinate system. k The x-coordinate and y-coordinate indicate the vehicle's position at time point k. k Let θ represent the y-coordinate indicating the vehicle's position at time point k, and let θ be the coordinate of the vehicle's position at time point k. k This indicates the rotation angle of the vehicle relative to a reference axis (e.g., the x-axis).

[0223] In equation 14, Let represent a unit vector, which has a unit size (e.g., 1) and a direction from the first vertex to the second vertex in the vehicle coordinate system at time k. Let represent the coordinates of the first vertex of the vehicle coordinate system at time point k, and This represents the coordinates of the second vertex of the vehicle coordinate system at time point k. The vector represents the coordinates of the second vertex in the vehicle coordinate system at time k, and the coordinates in the world coordinate system corresponding to the vehicle's position at time k.

[0224] For example, refer to Figure 8 The second geometric relationship information of the edge can be determined based on Equation 15 shown below.

[0225] Equation 15:

[0226]

[0227] In the non-restrictive example, in Equation 15, similar to Equation 4, in the vehicle coordinate system at time point k, and Let represent the slope and y-intercept of the line corresponding to the first edge (e.g., the left edge) of the k-th frame image, respectively. and y represents the slope and y-intercept of the line corresponding to the second side (e.g., the right edge) of the k-th frame image, respectively.

[0228] In Equation 15, similar to Equation 14, This represents a unit vector, which has a unit size (e.g., 1) and a direction from the first vertex to the second vertex according to the vehicle coordinate system at time point k.

[0229] Electronic devices can adjust the vehicle's positioning information based on the difference between the first and second geometric relationship information of the edges.

[0230] For example, electronic devices can adjust the vehicle's location information based on the following equation.

[0231] Equation 16:

[0232]

[0233]

[0234] Figure 9 Examples of determining first and second geometric relationship information of objects in or around a parking area according to one or more embodiments are shown.

[0235] refer to Figure 9 In a non-limiting example, electronic devices (e.g., Figure 10 The electronic device 1000 can detect objects in or around the parking area as landmarks (or candidate landmarks). The electronic device can determine the score of the object. (See reference above.) Figure 2 The electronic device can detect the object's score based on the angle between the vehicle and the parking area. Based on the object's score being less than a threshold score, the electronic device can determine the object's first geometric relationship information and second geometric relationship information.

[0236] The electronic device can determine the distance from the vehicle to the object from the first sensing data as first geometric relationship information. The electronic device can use a value based on the distance from the vertex of the parking area to the vehicle as the distance from the vehicle to the object. For example, as... Figure 9 As shown, when the length of one axis (e.g., the longitudinal axis) of the parking area is constant and the object is located at one end of that axis of the parking area, the value based on the distance from the vertex of the parking area to the vehicle can be related to the distance from the vehicle to the object.

[0237] For example, an electronic device can determine the first geometric relationship information corresponding to an object as shown in Equation 17 below.

[0238] Equation 17:

[0239]

[0240] In the non-restrictive example, in Equation 17, It is the first geometric relation information corresponding to the object, which is represented by the vector from the vehicle's position to the object along... The vector obtained by directional projection. Indicates and A vertical unit vector.

[0241] Similar to Equation 1, in Equation 17, This represents the x-coordinate of the second vertex (e.g., the right vertex) according to the world coordinate system, and This represents the y-coordinate of the second vertex in the world coordinate system.

[0242] Similar to Equation 7, in the world coordinate system, x k Let x represent the x-coordinate indicating the vehicle's position at time point k, and y represent the x-coordinate indicating the vehicle's position at time point k. k This represents the y-coordinate indicating the vehicle's position at time point k.

[0243] In the world coordinate system, This represents the vector from the second vertex (e.g., the right vertex) to the vehicle's position, and This indicates that the distance from the second vertex to the vehicle's position is along... The length is obtained by projecting the direction.

[0244] In the world coordinate system, This value is set based on the initial parking target point. For example, this value can be obtained by subtracting the radius of the vehicle wheels (e.g., the rear wheels) from the length along the longitudinal axis of the parking area.

[0245] The electronic device can determine the distance from the vehicle to the object from the second sensing data as second geometric relationship information. The electronic device can use a distance distribution to determine the distance from the vehicle to the object based on an image. For example, the electronic device can segment a region corresponding to the object from an image. The electronic device can determine the distance from each pixel included in the region corresponding to the object to the vehicle. The distance from the pixel to the vehicle can be determined along a specific direction (e.g., a vector). The distance projected (direction) of the object to the vehicle. An electronic device can classify multiple pixels based on their distances from the vehicle. For example, the electronic device can determine a histogram of multiple pixels based on the distance range from each pixel to the vehicle. The electronic device can determine a threshold frequency based on the maximum value of the histogram, and can determine the representative distance of the distance range with frequencies greater than or equal to the threshold frequency as the distance from the object to the vehicle.

[0246] For example, an electronic device can determine the second geometric relationship information corresponding to an object based on Equation 18 below.

[0247] Equation 18:

[0248]

[0249]

[0250] In the non-restrictive example, in Equation 18, It is the second geometric relation information corresponding to the object, which is represented by the vector from the vehicle position to the object along... The vector obtained by directional projection. Similar to Equation 17, Indicates and A vertical unit vector. γ is a value that can be set using the final stopping point and can be set to, for example, the radius of a vehicle wheel (e.g., the rear wheel).

[0251] In equation 18, H represents the distance from the vehicle to the object. k This represents a histogram that classifies multiple pixels within a region corresponding to an object based on the distance from the pixel to the vehicle. d represents the representative value for dividing the distance interval from the pixel to the vehicle. H k (d) represents the frequency of the distance interval corresponding to the representative value, and ε represents the coefficient set depending on the design.

[0252] Electronic devices can adjust vehicle positioning information based on the difference between the first and second geometric relationship information of an object.

[0253] For example, electronic devices can adjust the positioning information of an object as shown in Equation 19 below.

[0254] Equation 19:

[0255]

[0256] In the example, when multiple landmarks are selected as reference landmarks, the electronic device can adjust the positioning information based on multiple landmarks by repeatedly performing the operation of adjusting the positioning information based on each landmark. For example, when selecting multiple landmarks as reference landmarks from the k-th frame image at time point k, the electronic device can repeatedly adjust the positioning information at time point k based on the difference between the first geometric relationship information and the second geometric relationship information corresponding to each landmark. When the adjustment for multiple landmarks is completed, the electronic device can obtain the adjusted positioning information as the positioning information at time point k+1.

[0257] Figure 10 An example electronic device according to one or more embodiments is shown.

[0258] See Figure 10 In a non-limiting example, electronic device 1000 may include a sensing data acquisition unit 1010, a processor 1020, a memory 1030, and a communicator 1040.

[0259] The sensing data acquisition device 1010 can acquire sensing data. In the example, the sensing data acquisition device 1010 may include a sensor (e.g., a vision sensor or an inertial sensor). The sensing data acquisition device 1010 can generate sensing data based on sensing data collected from the sensor. For example, the sensing data acquisition device 1010 may include at least a portion of a communicator 1040. The sensing data acquisition device 1010 can be implemented as a device separate from the sensor and can receive sensing data from the sensor or a device including the sensor. For example, the sensing data acquisition device 1010 can acquire frame images based on a vision sensor. For example, the sensing data acquisition device 1010 can acquire change information of the vehicle's positioning information based on an inertial sensor.

[0260] In the example, processor 1020 can acquire frame images at each time point through sensing data acquisition unit 1010. Processor 1020 can select reference landmarks from the frame images. Processor 1020 can determine (e.g., update) the vehicle's positioning information by using the geometric relationship information of the reference landmarks with respect to the vehicle.

[0261] Processor 1020 can acquire first sensing data and second sensing data through sensing data acquisition unit 1010. Processor 1020 can determine the vehicle's positioning information using the first sensing data. Processor 1020 can determine the landmark scores in the second sensing data. Processor 1020 can determine first geometric relationship information of the landmarks from the first sensing data and can determine second geometric relationship information of the landmarks from the second sensing data. Processor 1020 can adjust the positioning information by using the difference between the determined first and second geometric relationship information.

[0262] The processor 1020 may also execute programs and / or control other operations or functions of the electronic device 1000 and the operation of the vehicle, and may include any one or a combination of two or more units, such as a central processing unit (CPU), a graphics processing unit (GPU), a neural processing unit (NPU), and a tensor processing unit (TPU), but is not limited to the examples above.

[0263] In the example, memory 1030 may temporarily and / or permanently store at least one of frame images, first sensing data, second sensing data, candidate landmarks, scores of candidate landmarks, reference landmarks (e.g., first reference landmark or second reference landmark), objects, vehicle location information, or geometric relationship information (e.g., first geometric relationship information or second geometric relationship information). Memory 1030 may store instructions for determining (e.g., adjusting or updating) vehicle location information, determining geometric relationship information, and / or selecting reference landmarks (e.g., first reference landmark or second reference landmark). However, the foregoing example is merely an example, and the information stored in memory 1030 is not limited thereto.

[0264] Memory 1030 may include computer-readable instructions. Processor 1020 may be configured to execute computer-readable instructions, such as those stored in memory 1030, and through the execution of these computer-readable instructions, processor 1020 is configured to perform one or more, or any combination thereof, of the operations and / or methods described herein. Memory 1030 may be volatile or non-volatile memory.

[0265] The communicator 1040 can send or receive one or more of the following: frame images, first sensing data, second sensing data, candidate landmarks, scores of candidate landmarks, reference landmarks (e.g., first reference landmark or second reference landmark), object or vehicle location information, or geometric relationship information (e.g., first geometric relationship information or second geometric relationship information). The communicator 1040 can establish wired and / or wireless communication channels with external devices (e.g., electronic devices and servers), and can communicate via long-range communication networks (such as cellular communication), short-range wireless communication, local area network (LAN) communication, Bluetooth, etc. TM Establish communication with external devices via Wi-Fi Direct or Infrared Data Association (IrDA), traditional cellular networks, fourth-generation (4G) and / or 5G networks, next-generation communications, the Internet, or computer networks (e.g., LANs or wide area networks (WANs)).

[0266] This article describes and this article is about Figures 1 to 10The disclosed electronic devices, processors, memories, neural networks, vehicles, electronic device 1000, sensing data acquisition device 1010, processor 1020, memory 1030, and communicator 1040 are implemented by or represent hardware components. As described above, or in addition to the above description, examples of hardware components that can be used to perform the operations described in this application, where appropriate, include controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more hardware components used to perform the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). A processor or computer may be implemented by one or more processing elements (e.g., logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field-programmable gate arrays, programmable logic arrays, microprocessors, or any other device or combination of devices configured to respond to and execute instructions in a defined manner to achieve a desired result). In one example, the processor or computer includes (or is connected to) one or more memories storing instructions or software executed by the processor or computer. Hardware components implemented by the processor or computer can execute instructions or software, such as an operating system (OS) and one or more software applications running on the OS, to perform the operations described in this application. Hardware components can also access, manipulate, process, create, and store data in response to the execution of instructions or software. For brevity, the singular terms "processor" or "computer" may be used in the description of the examples described in this application, but multiple processors or computers may be used in other examples, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component or two or more hardware components may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors or another processor and another controller. One or more processors or a processor and a controller may implement a single hardware component, or two or more hardware components. As described above, or in addition to the above description, the example hardware components may have any one or more different processing configurations, examples of which include a single processor, a standalone processor, a parallel processor, a single instruction single data (SISD) multiprocessing, a single instruction multiple data (SIMD) multiprocessing, multiple instruction single data (MISD) multiprocessing, and multiple instruction multiple data (MIMD) multiprocessing.

[0267] Perform the operations described in this application Figures 1 to 10The methods illustrated are executed by computing hardware, such as one or more processors or a computer, wherein the computing hardware is implemented as described above to implement instructions or software to perform the operations performed by these methods as described in this application. For example, a single operation or two or more operations may be executed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be executed by one or more processors or a processor and a controller, and one or more other operations may be executed by one or more other processors or another processor and another controller. One or more processors or a processor and a controller may execute a single operation or two or more operations.

[0268] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above can be written as computer programs, code segments, instructions, or any combination thereof, for individually or collectively instructing or configuring one or more processors or computers to operate as machines or special-purpose computers to perform the operations performed by the hardware components and methods described above. In one example, the instructions or software include machine code that is directly executed by one or more processors or computers, such as machine code generated by a compiler. In another example, the instructions or software include higher-level code that is executed by one or more processors or computers using an interpreter. The instructions or software can be written using any programming language based on the block diagrams and flowcharts shown in the accompanying drawings and the corresponding description herein (which discloses algorithms for performing the operations performed by the hardware components and the methods described above).

[0269] Instructions or software used to control computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above, along with any associated data, data files, and data structures, may be recorded, stored, or fixed in or on one or more non-transitory computer-readable storage media, and therefore are not signals in themselves. Examples of non-transitory computer-readable storage media, as described above or in addition to the above description, include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD- RLTH, BD-RE, Blu-ray or optical disc storage devices, hard disk drives (HDDs), solid-state drives (SSDs), flash memory, card-type memory (e.g., multimedia cards or microcards (e.g., Secure Digital (SD) or Extreme Digital (XD))), magnetic tape, floppy disks, magneto-optical data storage devices, optical data storage devices, hard disks, solid-state drives, and any other devices configured to: store instructions or software and any associated data, data files, and data structures in a non-transitory manner, and provide instructions or software and any associated data, data files, and data structures to a processor or computer such that one or more processors or computers can execute the instructions. In the example, the instructions or software and any associated data, data files, and data structures are distributed across a networked computer system, causing one or more processors or computers to store, access, and execute the instructions and software and any associated data, data files, and data structures in a distributed manner.

[0270] While this disclosure includes specific examples, it will be apparent upon understanding the disclosure of this application that various changes in form and detail may be made to these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein should be considered descriptive only and not for limiting purposes. The description of features or aspects in each example is intended to apply to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order and / or if components in the described system, architecture, device, or circuit are combined in a different manner and / or replaced or supplemented by other components or their equivalents.

[0271] Therefore, in addition to the foregoing disclosure and all the accompanying drawings, the scope of this disclosure also includes the claims and their equivalents, that is, all variations within the scope of the claims and their equivalents should be interpreted as being included within this disclosure.

Claims

1. A processor-implemented method, the method comprising: In the first frame image corresponding to the first time point, a first reference landmark is selected from a plurality of candidate landmarks related to the parking area of ​​the vehicle; The vehicle's positioning information is determined by using the geometric relationship information of the selected first reference landmark relative to the vehicle; In the second frame image corresponding to the second time point, a second reference landmark is selected from a second plurality of candidate landmarks, the second time point being later in time than the first time point; as well as The positioning information is updated using the geometric relationship information of the selected second reference landmark relative to the vehicle.

2. The method according to claim 1, wherein, Determining the vehicle's location information includes determining the vehicle's parking area. Each of the first and second frame images includes a region corresponding to a portion of the determined parking area, and The selection of the first reference landmark includes: Detect the first plurality of candidate landmarks based on the determined parking areas; and Calculate the scores of the first plurality of candidate landmarks detected.

3. The method according to claim 2, wherein, Determining the parking area includes: Based on the occupancy information of the space around the vehicle and the volume information of the vehicle, the parking area is determined from the area within the space around the vehicle that can be occupied by the vehicle.

4. The method according to claim 2, wherein, Detecting the first plurality of candidate landmarks includes detecting the vertices of the parking area as candidate landmarks, and Calculating the score of the first plurality of candidate landmarks includes calculating the score of the vertex based on the distance difference between the reference distance between reference vertices at the reference time point and the distance between the first vertices at the first time point.

5. The method according to claim 2, wherein, Detecting the first plurality of candidate landmarks includes detecting the edges of the parking area as candidate landmarks, and Calculating the score of the first plurality of candidate landmarks includes calculating the score of the edge based on the angle of the first edge in the first frame image.

6. The method according to claim 2, wherein, Detecting the first plurality of candidate landmarks includes detecting objects in or around the parking area as candidate landmarks, and Calculating the score of the first plurality of candidate landmarks includes calculating the score of the object based on the angle between the vehicle and the parking area in the first frame image.

7. The method according to claim 2, wherein, Selecting the first reference landmark includes selecting the first vertex of the parking area in the first frame image as the first reference landmark, and The selection of the second reference landmark includes selecting the second vertex and the second edge of the parking area in the second frame image as the second reference landmark.

8. The method according to claim 7, wherein, Updating the vehicle's location information also includes: Based on acquiring a third frame image corresponding to a third time point that is later than the second time point, the vertices of the parking area in the third frame image are excluded from the third reference landmark.

9. The method according to claim 2, wherein, Selecting the first reference landmark includes selecting the first edge of the parking area in the first frame image as the first reference landmark, and Selecting the second reference landmark includes selecting the second edge of the parking area in the second frame image and objects in or around the parking area in the second frame image as the second reference landmark.

10. The method according to claim 9, wherein, Updating the vehicle's location information also includes: Based on acquiring a third frame image corresponding to a third time point that is later than the second time point, the third edge of the parking area in the third frame image is excluded from the third reference landmark.

11. The method according to claim 1, wherein, Updating the vehicle's location information using the geometric relationship information of the second reference landmark includes: The vehicle's positioning information is updated based on the difference between the comparative geometric relationship information of the second reference landmark and the geometric relationship information.

12. The method according to claim 11, wherein, Updating the vehicle's location information using the geometric relationship information of the second reference landmark includes: The comparative geometric relationship information is obtained by transforming the determined positioning information using inertial data acquired from inertial sensors.

13. The method according to claim 11, wherein, Updating the vehicle's location information using the geometric relationship information of the second reference landmark includes: The comparative geometric relationship information is obtained by using a map that indicates the locations of the multiple candidate landmarks.

14. The method according to claim 1, wherein, Selecting the second reference landmark includes: determining whether to select a candidate landmark as the second reference landmark based on the result of comparing the score of each candidate landmark with a threshold score set for that candidate landmark, and The method of using the geometric relationship information of the second reference landmark to update the vehicle's positioning information includes: in response to not all of the second plurality of candidate landmarks being selected as the second reference landmark, skipping the update of the positioning information based on the second frame image.

15. A processor-implemented method, the method comprising: The vehicle's location information is determined by using first sensing data acquired from the first sensor; Based on the scores of landmarks in the second sensing data obtained from the second sensor, first geometric relationship information of the landmarks from the first sensing data and second geometric relationship information of the landmarks from the second sensing data are determined; as well as The positioning information is adjusted by using the difference between the determined first geometric relationship information and the second geometric relationship information.

16. The method according to claim 15, wherein, The first sensor includes an inertial sensor. The first sensing data includes changes in the vehicle's positioning information, and The method of determining the vehicle's location information by using the first sensing data includes: in response to acquiring the first sensing data, changing the vehicle's location information by using the change information to update the location information.

17. The method according to claim 15, wherein, The second sensor includes a vision sensor, and The determination of the first geometric relationship information and the second geometric relationship information includes: The image is acquired based on the second sensing data; Detect the landmark from the image; Calculate the scores of the landmarks detected from the image; and The second geometric relationship information is determined based on the score.

18. The method according to claim 15, wherein, Adjusting the location information includes: In response to the score being less than a threshold score, the location information is adjusted; and In response to the score being greater than or equal to the threshold score, the adjustment of the positioning information based on the difference between the first geometric relationship information and the second geometric relationship information is skipped.

19. The method according to claim 15, wherein, The landmarks include multiple candidate landmarks, and The determination of the first geometric relationship information and the second geometric relationship information includes: Calculate the scores of the plurality of candidate landmarks in the second sensing data; and Based on the scores of the multiple candidate landmarks, a reference landmark is determined among the multiple candidate landmarks, and The adjustment of the positioning information includes: adjusting the positioning information based on the difference between the first geometric relationship information and the second geometric relationship information corresponding to each reference landmark.

20. The method of claim 15, wherein, Determining the vehicle's location information also includes determining the vehicle's parking area, and The determination of the first geometric relationship information and the second geometric relationship information includes: detecting the landmark based on the determined parking area.

21. The method according to claim 20, wherein, Determining the parking area includes: Based on the space occupancy information and the vehicle's volume information, the parking area is determined within the area of ​​the space that can be occupied by the vehicle.

22. The method according to claim 20, wherein, Detecting the landmark includes detecting the vertices of the parking area as the landmark, and The determination of the first geometric relationship information and the second geometric relationship information includes: The score of the vertex is calculated based on the distance difference between the distance between vertices in the reference geometric relationship information obtained at the reference time point and the distance between vertices in the second sensing data; In response to a vertex's score being less than a vertex threshold score, the position of each vertex in the first sensing data is determined as the first geometric relationship information; and In response to the vertex's score being less than the vertex threshold score, the position of each vertex in the second sensing data is determined as the second geometric relationship information.

23. The method of claim 20, wherein, Detecting the landmark includes detecting the edge of the parking area as the landmark, and The determination of the first geometric relationship information and the second geometric relationship information includes: The score of the edge is calculated based on the angle of the edge in the second sensing data; In response to the edge score being less than an edge threshold score, the relative position of the edge to the vehicle and the angle between the edge and the vehicle in the first sensing data are determined as the first geometric relationship information; and In response to the edge score being less than the edge threshold score, the relative position of the edge to the vehicle and the angle between the edge and the vehicle in the second sensing data are determined as the second geometric relationship information.

24. The method of claim 20, wherein, Detecting the landmark includes detecting objects in or around the parking area as the landmark, and The determination of the first geometric relationship information and the second geometric relationship information includes: The score of the object is calculated based on the angle between the vehicle and the parking area in the second sensing data; In response to the object's score being less than an object threshold score, the distance from the vehicle to the object in the first sensing data is determined as the first geometric relationship information; and In response to the object's score being less than the object's threshold score, the distance from the vehicle to the object in the second sensing data is determined as the second geometric relationship information.

25. The method according to claim 15, wherein, Determining the first geometric relationship information and the second geometric relationship information includes: The first geometric relationship information is obtained by using a map that indicates the location of the landmark.

26. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method according to claim 1.

27. An electronic device, comprising: The processor is configured to execute instructions; as well as A memory that stores the instructions, wherein execution of the instructions configures the processor to: In the first frame image corresponding to the first time point, a first reference landmark is selected from multiple candidate landmarks related to the parking area of ​​the vehicle; The vehicle's positioning information is determined by using the geometric relationship information between the selected first reference landmark and the vehicle. In the second frame image corresponding to a second time point that is later than the first time point, a second reference landmark is selected from multiple candidate landmarks; as well as The vehicle's positioning information is updated by using the geometric relationship information between the selected second reference landmark and the vehicle.

28. An electronic device, comprising: The processor is configured to execute instructions; as well as A memory that stores the instructions, wherein execution of the instructions configures the processor to: The vehicle's location information is determined by using first sensing data acquired from the first sensor; Based on the landmark scores in the second sensing data acquired from the second sensor, first geometrical relationship information of the landmark from the first sensing data and second geometrical relationship information of the landmark from the second sensing data are determined; and The positioning information is adjusted by using the difference between the determined first geometric relationship information and the second geometric relationship information.

Images