Parking space reconstruction method and device, electronic equipment and storage medium

Abstract

Embodiments of the present disclosure disclose a parking space reconstruction method and device, electronic equipment and a storage medium, wherein the method comprises: determining current parking space observation information in a vehicle coordinate system corresponding to a current target frame set, the current target frame set comprising a current frame and a preset number of historical frames; determining a current parking space tracking result based on the current parking space observation information; determining a first state quantity corresponding to each parking space of the current target frame set based on the current parking space tracking result; optimizing each first state quantity based on the current parking space observation information, the current parking space tracking result and a preset optimization rule, to obtain a second state quantity corresponding to each parking space of the current target frame set after optimization; and determining target position information in a world coordinate system corresponding to each parking space based on the second state quantity corresponding to each parking space of the current target frame set. The embodiments of the present disclosure realize camera image-based parking space reconstruction, without the need for laser radar, effectively reducing the cost of parking space reconstruction.

Description

Technical Field

[0001] This disclosure relates to computer vision technology, and in particular to a parking space reconstruction method, apparatus, electronic device, and storage medium. Background Technology

[0002] Parking space reconstruction has become a crucial technology in parking scenarios such as HPA (Hyper-Learning Parking) and AVP (Valet Parking), and is an indispensable part of high-precision maps. Current parking space reconstruction typically uses LiDAR scanning to obtain point clouds, and then extracts parking space parameters based on these point clouds to achieve reconstruction. However, this existing method of parking space reconstruction is costly. Summary of the Invention

[0003] To address the aforementioned technical problems, such as the high cost of parking space reconstruction, this disclosure is proposed. Embodiments of this disclosure provide a parking space reconstruction method, apparatus, electronic device, and storage medium.

[0004] According to one aspect of the present disclosure, a parking space reconstruction method is provided, comprising: determining current parking space observation information in a vehicle coordinate system corresponding to a current target frame set, the current target frame set including a current frame and a preset number of historical frames; determining a current parking space tracking result based on the current parking space observation information; determining a first state quantity corresponding to each parking space in the current target frame set based on the current parking space tracking result, the first state quantity including parking space information of the corresponding parking space in a world coordinate system; optimizing each of the first state quantities based on the current parking space observation information, the current parking space tracking result and a preset optimization rule to obtain optimized second state quantities corresponding to each parking space in the current target frame set; and determining target position information in a world coordinate system corresponding to each parking space based on the second state quantities corresponding to each parking space in the current target frame set.

[0005] According to another aspect of the present disclosure, a parking space reconstruction device is provided, comprising: a first determining module, configured to determine current parking space observation information in a vehicle coordinate system corresponding to a current target frame set, the current target frame set including a current frame and a preset number of historical frames; a first processing module, configured to determine a current parking space tracking result based on the current parking space observation information; a second determining module, configured to determine a first state quantity corresponding to each parking space in the current target frame set based on the current parking space tracking result, the first state quantity including parking space information of the corresponding parking space in a world coordinate system; a second processing module, configured to optimize each first state quantity based on the current parking space observation information, the current parking space tracking result and a preset optimization rule to obtain an optimized second state quantity corresponding to each parking space in the current target frame set; and a third processing module, configured to determine target position information in a world coordinate system corresponding to each parking space based on the second state quantity corresponding to each parking space in the current target frame set.

[0006] According to another aspect of the present disclosure, a computer-readable storage medium is provided, the storage medium storing a computer program for performing the parking space reconstruction method described in any of the above embodiments of the present disclosure.

[0007] According to another aspect of the present disclosure, an electronic device is provided, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the parking space reconstruction method according to any of the above embodiments of the present disclosure.

[0008] Based on the parking space reconstruction method, apparatus, electronic device and storage medium provided in the above embodiments of this disclosure, the state variables of the parking space in the world coordinate system are optimized by using the observation information, tracking results and certain optimization rules of the parking space, thereby realizing parking space reconstruction based on camera images. This eliminates the need for LiDAR, effectively reduces the cost of parking space reconstruction, and solves the problem of high cost of parking space reconstruction in the prior art.

[0009] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0010] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0011] Figure 1 This is an exemplary application scenario of the parking space reconstruction method provided in this disclosure;

[0012] Figure 2 This is a schematic flowchart of a parking space reconstruction method provided in an exemplary embodiment of this disclosure;

[0013] Figure 3 This is a flowchart illustrating a parking space reconstruction method provided in another exemplary embodiment of this disclosure;

[0014] Figure 4 This is a flowchart illustrating step 204 provided in an exemplary embodiment of this disclosure;

[0015] Figure 5 This is a schematic diagram illustrating the principle of determining the coordinates of the third and fourth endpoints of the entry line provided in an exemplary embodiment of this disclosure;

[0016] Figure 6 This is a flowchart illustrating step 204 provided in another exemplary embodiment of this disclosure;

[0017] Figure 7 This is a flowchart illustrating a parking space reconstruction method provided in yet another exemplary embodiment of this disclosure;

[0018] Figure 8 This is a schematic diagram of the structure of a parking space reconstruction device provided in an exemplary embodiment of this disclosure;

[0019] Figure 9 This is a schematic diagram of the structure of the second processing module 504 provided in an exemplary embodiment of this disclosure;

[0020] Figure 10 This is a schematic diagram of the structure of the first processing unit 5041 provided in an exemplary embodiment of this disclosure;

[0021] Figure 11 This is a schematic diagram of the structure of the third processing unit 5043 provided in an exemplary embodiment of this disclosure;

[0022] Figure 12 This is a schematic diagram of the structure of a parking space reconstruction device provided in another exemplary embodiment of this disclosure;

[0023] Figure 13 This is a schematic diagram of the structure of the third processing module 505 provided in an exemplary embodiment of this disclosure;

[0024] Figure 14 This is a schematic diagram of the structure of a parking space reconstruction device provided in yet another exemplary embodiment of the present disclosure;

[0025] Figure 15This is a schematic diagram of the structure of the first determining module 501 provided in an exemplary embodiment of this disclosure;

[0026] Figure 16 This is a schematic diagram of the structure of the first processing module 502 provided in an exemplary embodiment of this disclosure;

[0027] Figure 17 This is a schematic diagram of the structure of the second determining module 503 provided in an exemplary embodiment of this disclosure;

[0028] Figure 18 This is a schematic diagram of the structure of an application embodiment of the electronic device disclosed herein. Detailed Implementation

[0029] Hereinafter, exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. It is obvious that the described embodiments are merely some embodiments of the present disclosure, and not all embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0030] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0031] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0032] It should also be understood that in the embodiments disclosed herein, "multiple" can refer to two or more, and "at least one" can refer to one, two or more.

[0033] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0034] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0035] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0036] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0037] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0038] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0039] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0040] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0041] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0042] This disclosure outlines

[0043] In developing this disclosure, the inventors discovered that parking space reconstruction has become a crucial technology in parking scenarios such as HPA (Hyper-Learning Parking) and AVP (Valet Parking), and is an indispensable part of high-precision maps. Current parking space reconstruction typically uses LiDAR scanning to obtain point clouds, and then extracts parking space parameter information based on the point cloud to achieve reconstruction. However, this existing parking space reconstruction method is costly.

[0044] Exemplary Overview

[0045] Figure 1 This is an exemplary application scenario of the parking space reconstruction method provided in this disclosure.

[0046] In a parking scenario, a camera mounted on the vehicle captures images of the surrounding environment, including parking space information. Based on these images, the current parking space observation information in the vehicle coordinate system corresponding to the current target frame set can be determined. The current target frame set includes the current frame and a preset number of historical frames. The current parking space observation information can include one or more of the following: parking space type, parking space corner position, parking space long side direction, parking space occupancy status, and parking space corner type (whether it is a cutoff point). Then, based on this current parking space observation information, the current parking space tracking result is determined. The result can include tracking information for all parking spaces present in the current target frame set, as well as tracking information for parking spaces that existed in the previous target frame set but not in the current target frame set. The previous target frame set includes the frame preceding the current frame and a preset number of historical frames prior to that frame. The parking space tracking information can include parking space identifiers (such as parking space IDs), the number of times the parking space has been observed, etc. Specifically, the parking spaces existing in the current target frame set include those that existed in a preset number of historical frames but not in the current frame, and newly appearing parking spaces in the current frame. Based on the current parking space tracking results, a first state variable is determined for each parking space in the current target frame set. This first state variable includes the parking space information in the world coordinate system. For parking spaces already existing in a preset number of historical frames, the optimized state variable from the previous target frame set can be used as the first state variable for that parking space in the current target frame set. For newly observed parking spaces in the current frame, the initial state variable for that parking space can be determined based on its observation information and used as its first state variable. The first state variables for the same parking space in different frames are the same. Based on the current parking space observation information, the current parking space tracking results, and the preset optimization rules, each first state variable is optimized to obtain the optimized second state variable corresponding to each parking space in the current target frame set. The preset optimization rules can be set according to actual needs. Specifically, the objective function can be set based on adjacent parking space constraints, parking space entry line endpoint observation reprojection constraints, etc. The optimization result makes the objective function value of each parking space minimized under certain constraints. The optimized second state variable is used to determine the target position information in the world coordinate system corresponding to each parking space, which not only ensures the accuracy of parking space information, but also makes the parking space satisfy the constraints of adjacent parking spaces. Parking space reconstruction is realized based on the observation and tracking of parking spaces, thus realizing parking space reconstruction based on camera images without the need for LiDAR, effectively reducing the cost of parking space reconstruction.

[0047] Exemplary methods

[0048] Figure 2 This is a schematic flowchart of a parking space reconstruction method provided in an exemplary embodiment of this disclosure. This embodiment can be applied to electronic devices, specifically, for example, on an in-vehicle computing platform. Figure 2 As shown, it includes the following steps:

[0049] Step 201: Determine the current parking space observation information in the vehicle coordinate system corresponding to the current target frame set. The current target frame set includes the current frame and a preset number of historical frames.

[0050] The current target frame set is the target frame set corresponding to the current moment. It includes the current frame and a preset number of historical frames, which can be set according to actual needs. For example, it can be set to 7 frames, in which case the current target frame set includes 8 frames. The preset number can also be determined based on the vehicle's travel distance; for example, the number of frames corresponding to a distance of 5 meters minus the current frame is the preset number. There is no specific limitation. The current parking space observation information can include parking space-related information obtained from each frame in the current target frame set, such as one or more of the following: parking space type, parking space corner position, parking space long side direction, parking space occupancy status, and parking space corner type (whether it is a cutoff point). The current parking space observation information can be obtained based on environmental image data captured by a camera on the vehicle. For example, the environmental image data can be converted to the vehicle coordinate system to obtain a bird's-eye view image in the vehicle coordinate system, and the current parking space observation information can be determined based on the bird's-eye view image. Alternatively, the first parking space observation information can be obtained based on the environmental image data, and then the first parking space observation information can be converted to the vehicle coordinate system to obtain the current parking space observation information. There is no limitation on the specific method of obtaining the current parking space observation information.

[0051] Step 202: Determine the current parking space tracking result based on the current parking space observation information.

[0052] The current parking space tracking result is obtained by tracking parking spaces. This result can include tracking information for each parking space present in the current target frame set, as well as tracking information for parking spaces that existed in the previous target frame set but not in the current target frame set. The previous target frame set includes the frame preceding the current frame and a preset number of historical frames prior to that frame. The parking space tracking information can include parking space identifiers (e.g., parking space ID), the number of times the parking space has been observed, etc. Specifically, the parking spaces present in the current target frame set include those that existed in a preset number of historical frames but not in the current frame, and newly appearing parking spaces in the current frame. The parking space observation count refers to the number of times that parking space is observed in the current target frame set, that is, the number of frames in the current target frame set that track that parking space. For example, if the current target frame set includes 8 frames, and parking space 1 is tracked in the first 4 frames, then the observation count for parking space 1 is 4. As the vehicle moves, new frame data is continuously generated. Each new frame becomes the current frame, and the original current frame becomes a frame in a preset number of historical frames. The earliest frame in the original preset number of historical frames is eliminated, forming a new current target frame set. If parking space 1 is not observed in the current frame of the new current target frame set, then the observation count for parking space 1 is reduced by 1 to 3 due to the elimination of the earliest historical frame. For a newly observed parking space in the current frame (such as parking space 2), its observation count is set to 1. When parking space 2 is tracked again in the next frame, the observation count for parking space 2 becomes 2. And so on, maintaining the current parking space tracking results of the current target frame set in real time. Parking space tracking can be achieved using a preset tracking algorithm. This preset algorithm can employ any feasible algorithm based on actual needs, such as the intersection-over-union (IoU) ratio of parking spaces in two adjacent frames. This determines the correspondence between parking spaces in the two frames, and then, based on the tracking result of the previous frame, determines the tracking result of the parking space in the current frame, and so on, to achieve parking space tracking. The specific settings can be configured according to actual requirements.

[0053] In practical applications, the current target frame set can be updated over time or as the vehicle moves, using a sliding window algorithm or other feasible methods, without any specific limitations.

[0054] Step 203: Based on the current parking space tracking results, determine the first state quantity corresponding to each parking space in the current target frame set. The first state quantity includes the parking space information of the corresponding parking space in the world coordinate system.

[0055] Among them, the first state quantity is the initial state quantity of each parking space in the optimization process of the current target frame set. The first state quantity of each parking space can be set according to actual needs.

[0056] For example, for a parking space that already exists in the parking space tracking results corresponding to the previous target frame set, the second state quantity determined based on the previous target frame set is used as the first state quantity of the parking space in the current target frame set. For a newly added parking space in the current parking space tracking results, the first state quantity of the parking space is determined based on the parking space observation information corresponding to the parking space in the current parking space observation information. The first state quantity may specifically include information such as the x-coordinate and y-coordinate of the center point of the parking space entry line, the yaw angle of the entry line, and the width of the entry line, which can be set according to actual needs.

[0057] Step 204: Based on the current parking space observation information, the current parking space tracking results and the preset optimization rules, optimize each first state variable to obtain the optimized second state variable corresponding to each parking space in the current target frame set.

[0058] The preset optimization rules can be set according to actual needs. For example, the objective function can be set based on adjacent parking space constraints, parking space entry line endpoint observation reprojection constraints, etc. The optimization result is to minimize the objective function value for each parking space under certain constraints. Specific details are not limited. The second state variable is similar to the first state variable, including the optimized parking space information in the world coordinate system for its corresponding parking space. Further details will not be elaborated further.

[0059] In practical applications, the optimization of the first state variable can be achieved using an optimizer. The optimizer can be any implementable optimizer, and this disclosure does not impose any restrictions.

[0060] Step 205: Based on the second state variables corresponding to each parking space in the current target frame set, determine the target position information in the world coordinate system corresponding to each parking space.

[0061] The target location information of the parking space in the world coordinate system can include the coordinates of the parking space's outline points, such as at least the coordinates of its four corner points, which can be set according to actual needs. For a single parking space, its target location information is determined by a second state variable after at least one optimization process, which can also be set according to actual needs.

[0062] For example, the obtained second state variables include the x-coordinate and y-coordinate of the center point of the optimized entry line of the parking space, the yaw angle of the entry line, and the width of the entry line. Based on the yaw angle of the entry line and the parking space type, the direction of the parking space can be determined. Based on the width of the entry line, the length of the adjacent side of the entry line can be determined. Based on the x-coordinate and y-coordinate of the center point of the entry line, the yaw angle of the entry line, and the length of the entry line, the coordinates of the two ends of the entry line can be determined. The coordinates of the two ends of the entry line are the coordinates of the two corner points of the parking space. Based on the coordinates of the two ends of the entry line, the parking space type, and the length of the adjacent side of the entry line, the coordinates of the other two corner points of the parking space can be determined, thereby obtaining the target location information of the parking space.

[0063] For example, when a parking space exists in the previous target frame set but not in the current target frame set, that is, when the observation of the parking space in the current target frame set ends, the target position information of the parking space in the world coordinate system can be determined based on the second state quantity of the parking space obtained from the previous target frame set. When a parking space can still be observed in the current target frame set, the next optimization process is entered based on the second state quantity of the parking space, which serves as the first state quantity of the parking space in the current target frame set for the next optimization process. Optimization is performed based on this until the parking space is slid out of the current target frame set, and the target position information of the parking space can be determined.

[0064] Optionally, parking spaces can be perpendicular, lateral, or angled. The length of the entry line for a perpendicular parking space is the length of its shorter side. The length of the entry line for a lateral parking space is the length of its longer side. Angled parking spaces are similar to perpendicular parking spaces, with the entry line being the length of the shorter side. For angled parking spaces, the target position information can be determined by combining the angle between adjacent sides or the direction of the longer side. For example, when a parking space slides out of the current target frame set, the final second state variable of that parking space is [x1, y1, yaw1, width1]. The length of the side adjacent to the entry line can be a preset length or a length based on observation results. For the preset length, different lengths can be set for different types of parking spaces. For example, for a lateral parking space, if the entry line is the longer side, the width is 2.5 meters; for a perpendicular parking space, if the entry line is the shorter side, the longer side is 5.3 meters. For angled parking spaces, the length can be set according to normal conditions or determined by observation results. The specific setting can be based on actual needs.

[0065] For example, by updating the current target frame set based on a sliding window, the target location information of the parking space can be determined when the parking space slides out of the current window.

[0066] The parking space reconstruction method provided in this embodiment optimizes the state variables of the parking space in the world coordinate system based on the observation information, tracking results and certain optimization rules of the parking space, thereby realizing parking space reconstruction based on camera images. It eliminates the need for LiDAR, effectively reduces the cost of parking space reconstruction, and solves the problem of high cost of parking space reconstruction in existing technologies.

[0067] Figure 3 This is a flowchart illustrating a parking space reconstruction method provided in another exemplary embodiment of this disclosure.

[0068] In an optional example, step 204, which optimizes each first state variable based on the current parking space observation information, the current parking space tracking result, and preset optimization rules to obtain the optimized second state variable corresponding to each parking space in the current target frame set, may specifically include the following steps:

[0069] Step 2041: Based on each first state variable and the vehicle pose corresponding to each frame in the current target frame set, determine the coordinates of the first endpoint and the second endpoint of the entry line corresponding to each parking space in the current target frame set in the vehicle coordinate system.

[0070] The vehicle pose corresponding to each frame refers to the vehicle's pose in the world coordinate system at the time the data for each frame is collected. For example, the vehicle pose corresponding to the current frame is the vehicle pose in the vehicle coordinate system at the time the data for the current frame is collected. Since the vehicle is constantly moving, there is a time interval between two frames, so the vehicle poses corresponding to different frames may be different. The first state variable of the parking space includes the x-coordinate and y-coordinate of the center point of the parking space's entry line, the yaw angle of the entry line, the width of the entry line, and other information. Combining the vehicle pose corresponding to each frame, the first and second endpoint coordinates of the parking space's entry line in the vehicle coordinate system can be deduced. For example, the endpoint coordinates of the parking space's entry line in the world coordinate system can be determined first based on the first state variable. Then, based on the transformation relationship between the world coordinate system and the vehicle coordinate system, the endpoint coordinates of the entry line in the world coordinate system can be transformed to the vehicle coordinate system of the corresponding frame, thereby obtaining the first and second endpoint coordinates of the parking space's entry line in the vehicle coordinate system of its corresponding frame.

[0071] Step 2042: Based on the current parking space observation information, determine the coordinates of the first and second observation endpoints of the entry line corresponding to each parking space in the current target frame observation.

[0072] The coordinates of the first and second observation endpoints are based on the coordinates of the two endpoints of the parking space entry line obtained from observations. Since the current parking space observation information includes one or more of the following: parking space type, parking space corner location, parking space long side direction, parking space occupancy status, and parking space corner type (whether it is a cutoff point): the first and second observation endpoints of the entry line represent the two corners of the parking space. Therefore, the coordinates of the first and second observation endpoints can be obtained from the parking space corner location in the current parking space observation information.

[0073] Step 2043: Based on the first endpoint coordinates, second endpoint coordinates, first observation endpoint coordinates, and second observation endpoint coordinates corresponding to each parking space in the current target frame set, and the preset objective function, optimize each first state variable to obtain each second state variable.

[0074] The preset objective function can be set according to actual optimization needs. For example, the preset objective function may include the reprojection residual function of the parking line endpoints and the constraint residual function of adjacent parking spaces. The reprojection residual function of the parking line endpoints is used to determine the residuals between the coordinates of the first and second endpoints of each parking space in the current target frame set and the corresponding coordinates of the first and second observation endpoints. The constraint residual function of adjacent parking spaces is used to determine the constraint residuals of adjacent parking spaces in the state variables. The objective function value is determined by combining the endpoint reprojection residuals and the constraint residuals of adjacent parking spaces. The optimization objective is to minimize the objective function value under certain constraints on the state variables. The specific optimization principle will not be elaborated further.

[0075] This disclosure uses the first state variables of each parking space to deduce the coordinates of the entry and exit line endpoints, compares them with the observed coordinates of the entry and exit line endpoints, and optimizes the first state variables based on a preset objective function to achieve parking space reconstruction. This enables parking space reconstruction based on camera images, eliminating the need for LiDAR, effectively reducing parking space reconstruction costs, and solving the problem of high parking space reconstruction costs in existing technologies.

[0076] Figure 4 This is a flowchart illustrating step 204 provided in an exemplary embodiment of this disclosure.

[0077] In an optional example, the first state variable includes the coordinates of the center point of the parking space's entry line in the world coordinate system, the heading angle of the entry line, and the length of the entry line; step 2041, based on each first state variable and the vehicle pose corresponding to each frame in the current target frame set, determines the coordinates of the first endpoint and the second endpoint of the entry line corresponding to each parking space in the current target frame set in the vehicle coordinate system, including:

[0078] Step 20411: Based on the length of the entry line, the heading angle of the entry line, and the coordinates of the center point of the entry line in each of the first state variables, determine the coordinates of the third endpoint and the fourth endpoint of the entry line corresponding to each parking space in the current target frame set in the world coordinate system.

[0079] Parking spaces can be perpendicular, lateral, or angled. The length of the entry line for a perpendicular parking space is the length of its shorter side, and the length of the entry line for a lateral parking space is the length of its longer side. The first state variable is the parking space information in the world coordinate system. Therefore, based on the length of the entry line, the heading angle of the entry line, and the coordinates of the center point of the entry line in the first state variable, the coordinates of the two endpoints of the parking space's entry line can be deduced.

[0080] For example, Figure 5 This is a schematic diagram illustrating the principle of determining the coordinates of the third and fourth endpoints of the inbound line according to an exemplary embodiment of this disclosure. The first state variable is represented as [x1, y1, yaw1, width1], the coordinates of the center point of the inbound line are (x1, y1), the heading angle of the inbound line is yaw1, the length of the inbound line is width1, the coordinates of the third endpoint (x3, y3) correspond to the third endpoint P3, and the coordinates of the fourth endpoint (x4, y4) correspond to the fourth endpoint P4. The coordinates of the third and fourth endpoints can be determined as follows:

[0081]

[0082]

[0083]

[0084]

[0085] Step 20412: Based on the third endpoint coordinates and fourth endpoint coordinates corresponding to each parking space, and the vehicle poses corresponding to each frame in the current target frame set, determine the first endpoint coordinates and second endpoint coordinates corresponding to each parking space in the vehicle coordinate system of its corresponding frame.

[0086] Specifically, based on the vehicle pose corresponding to each frame, the coordinates of the third and fourth endpoints of each parking space are transformed into the vehicle coordinate system to obtain the coordinates of the first and second endpoints of each parking space in the vehicle coordinate system of its corresponding frame. The specific transformation principle will not be elaborated here.

[0087] In an optional example, step 2043 optimizes each first state variable based on the first endpoint coordinates, second endpoint coordinates, first observation endpoint coordinates, and second observation endpoint coordinates corresponding to each parking space in the current target frame set, and a preset objective function, to obtain each second state variable, including:

[0088] Step 20431: Determine the target function value based on the first endpoint coordinates, second endpoint coordinates, first observation endpoint coordinates, and second observation endpoint coordinates corresponding to each parking space in the current target frame set, as well as the preset target function.

[0089] The preset objective function can include the reprojection residual function of the entry line endpoints and the constraint residual function of adjacent parking spaces. The reprojection residual function of the entry line endpoints is used to determine the residuals between the coordinates of the first and second endpoints of each parking space in the current target frame set and the corresponding coordinates of the first and second observation endpoints. The constraint residual function of adjacent parking spaces is used to determine the constraint residuals of adjacent parking spaces in the state variables. The objective function value is determined by combining the endpoint reprojection residuals and the constraint residuals of adjacent parking spaces. The optimization objective is to minimize the objective function value under certain constraints on the state variables. The specific optimization principle will not be elaborated further.

[0090] Step 20432: Based on the objective function value, the least squares algorithm is used to update each first state variable to obtain the third state variable corresponding to each first state variable.

[0091] The objective function value represents the cost of the optimization result. Based on the objective function value, gradient descent is performed using the least squares algorithm to obtain the optimized result. The specific principle will not be elaborated further.

[0092] Step 20433: In response to each third state variable not meeting the preset conditions, each third state variable is treated as a first state variable and optimized again. This process is repeated until each third state variable meets the preset conditions, and then each third state variable is treated as a second state variable.

[0093] The preset conditions can be set according to actual needs. For example, the preset condition can be to minimize the objective function value, or it can be to minimize the objective function value under certain constraints. There are no specific limitations.

[0094] In practical applications, state variables can be optimized using optimizers. Solving optimization problems using optimizers can include: constructing a cost function, i.e., the objective function; constructing the optimization problem to be solved using the cost function; configuring the solver parameters and solving the problem, i.e., setting how to solve the problem and whether to output the solution process. Specific optimization tools can be used to solve the optimization problem, which will not be elaborated upon here.

[0095] Figure 6 This is a flowchart illustrating step 204 provided in another exemplary embodiment of this disclosure.

[0096] In an optional example, step 2042, based on the current parking space observation information, determines the coordinates of the first and second observation endpoints of the entry line corresponding to each parking space observed in the current target frame set, including:

[0097] Step 20421: Based on the current parking space observation information, determine the coordinates of the observation center point, the first observation endpoint, and the second observation endpoint of the parking space corresponding to each parking space in the current target frame observation.

[0098] The coordinates of the observation center point of the inflow line can be obtained based on the coordinates of the first and second observation endpoints, which will not be elaborated further.

[0099] Step 20431 involves determining the target function value based on the first endpoint coordinates, second endpoint coordinates, first observation endpoint coordinates, and second observation endpoint coordinates corresponding to each parking space in the current target frame set, as well as a preset target function. This includes:

[0100] Step 204311: Based on the first endpoint coordinates, second endpoint coordinates, first observation endpoint coordinates and second observation endpoint coordinates corresponding to each parking space in the current target frame set, determine the endpoint reprojection residual function value in the preset objective function.

[0101] The endpoint reprojection residual function value is obtained by comparing the observed endpoint coordinates (including the first and second observed endpoint coordinates) of the parking space's entry line with the endpoint coordinates (including the first and second endpoint coordinates) derived from the first state variable.

[0102] Step 204312: Based on the coordinates of the observation center point corresponding to each parking space in the current target frame set and the length of the entry line in the first state variable corresponding to each parking space, determine the value of the adjacent parking space constraint residual function in the preset objective function.

[0103] For example, the preset objective function is:

[0104]

[0105] Where e represents the number of frames included in the current target frame set, f represents the number of parking spaces in the i-th frame, and e ij Let m and n represent the endpoint reprojection residual of the j-th parking space in the i-th frame, and e represent two adjacent parking spaces in the adjacent parking space set. mn This represents the adjacent parking space constraint residuals of adjacent parking spaces m and n.

[0106] Among them, e ij It is expressed as follows:

[0107] e ij =z ij -h(T i ,P j )

[0108] Among them, z ij Indicates the vehicle's position Ti Observation parking space P j The obtained coordinates of the inflow line endpoints, namely the first and second observation endpoint coordinates mentioned above, h(T) i ,P j ) represents the coordinates of the inbound line endpoints obtained by back-calculation using the first state variable, namely the first endpoint coordinates and the second endpoint coordinates mentioned above.

[0109] e mn It is expressed as follows:

[0110] e mn =z mn -f(P m ,P n )

[0111] Among them, z mn Indicates adjacent parking space P m and P n The distance between the observation center points of the inflow line, f(P) m ,P n () represents the parking space P obtained by back-calculation based on the first state variable. m and P n The distance between the center points of the entry line.

[0112]

[0113] Among them, width m and width n These represent parking spaces P. m and P n The length of the inbound line in the first state variable.

[0114] Step 204313: Determine the objective function value based on the endpoint reprojection residual function value and the adjacent parking space constraint residual function value.

[0115] The objective function value can be determined based on the endpoint reprojection residual function value and the adjacent parking space constraint residual function value.

[0116] In practical applications, when determining the first state variables of each parking space, the heading angles of the entry lines of adjacent parking spaces can be set to the same heading angle based on the constraints of adjacent parking spaces, such as the adjacent parking spaces P mentioned above. m and P n In its first state variable, the heading angle yaw of the inbound line m =yaW n This ensures that the direction of the entry line of adjacent parking spaces in the reconstructed parking space results is consistent, which is more in line with the actual parking space situation.

[0117] Figure 7This is a schematic flowchart of a parking space reconstruction method provided in another exemplary embodiment of this disclosure.

[0118] In an optional example, after determining the current parking space tracking result based on the current parking space observation information in step 202, the method further includes:

[0119] Step 202a: Based on the current parking space observation information, the current parking space tracking results, and the preset grouping rules, determine the adjacent parking space groups in the current target frame set.

[0120] The preset grouping rules can be set according to actual needs, such as the distance between the entry corners of two parking space frames being less than a preset threshold, the two parking space frames having parallel sides, etc. Based on the preset grouping rules, a clustering algorithm is used to group adjacent parking spaces in the current target frame set.

[0121] Step 204 optimizes each first state variable based on the current parking space observation information, the current parking space tracking result, and preset optimization rules to obtain the optimized second state variables corresponding to each parking space in the current target frame set, including:

[0122] Step 2041a: Based on the current parking space observation information, the current parking space tracking result, the adjacent parking space grouping, the preset adjacent parking space constraint rules and the preset optimization rules, optimize each first state variable to obtain the second state variable corresponding to each parking space in the current target frame set.

[0123] The preset adjacent parking space constraint rules can include distance constraints and angle constraints. The distance constraint means that the distance between the center points of the entry lines of two parking spaces observed during the optimization process is half of the sum of the entry line lengths of the two parking spaces in the state variables. The angle constraint refers to the parallelism of adjacent parking spaces, that is, the entry lines of adjacent parking spaces should be parallel. Therefore, the entry lines of adjacent parking spaces in the state variables are optimized using the same heading angle.

[0124] This disclosure improves the accuracy of parking space reconstruction results by setting adjacent parking space constraint rules, making the reconstructed parking spaces more consistent with the actual adjacent parking space situation.

[0125] In an optional example, step 205, based on the second state variables corresponding to each parking space in the current target frame set, determines the target position information in the world coordinate system corresponding to each parking space, including:

[0126] Step 2051: Determine the next target frame set based on the current target frame set and the next frame.

[0127] The next target frame set is obtained by removing the oldest historical frame from the current target frame set and adding the new frame (i.e., the frame following the current frame). This can be implemented using a sliding window or other similar methods; the specific principles will not be elaborated further.

[0128] Step 2052: Take the next target frame set as the current target frame set. In response to the observation count of the target parking space in the parking space tracking result corresponding to the current target frame set being 0, determine the contour point coordinates of the target parking space in the world coordinate system based on the second state variable corresponding to the target parking space, and use it as the target position information of the target parking space.

[0129] Specifically, when a new frame is generated, time progresses, the next frame becomes the current frame, and the previous current frame becomes a history frame, and so on. The details will not be elaborated further. Taking a sliding window as an example, the current target frame set represents the current window. The current parking space tracking result records the parking space tracking information within the current window, as well as the parking space tracking information just after sliding out of the window. The parking space tracking information includes the parking space identifier and the number of observations. When the number of observations becomes 0, it indicates that the parking space has slid out of the window, which means that the optimization of that parking space is complete. The target location information of that parking space can be determined based on the optimization results.

[0130] In an optional example, when parking space 1 slides out of the window, its state variable no longer changes. If parking space 2, which is still in the window, is an adjacent parking space of parking space 1 that slid out of the window, then parking space 2 needs to maintain the adjacent parking space constraint with parking space 1 that slid out of the window. For example, the heading angle of the entry line of parking space 2 does not change during the optimization process and remains the same as the final optimized heading angle of parking space 1, and so on.

[0131] Step 2053: For parking spaces in the current target frame set whose observation count is greater than 0 in the parking space tracking results, the second state variable corresponding to the parking space is used as the first state variable, and optimization is continued until the observation count of the parking space is 0, thereby obtaining the target position information of the parking space.

[0132] For parking spaces with more than 9 observations in the current target frame, it means that the optimization has not been completed and further optimization is needed. The second state quantity after optimization in the previous optimization process is used as the initial state quantity of the current optimization process, that is, the first state quantity. The optimization continues according to the aforementioned optimization process until the observation count of the parking space is 0. The target location information of the parking space can be determined based on the second state quantity obtained by the final optimization.

[0133] In one optional example, the current parking space tracking result includes the number of observations for each tracked parking space; after determining the current parking space tracking result based on the current parking space observation information in step 202, the method further includes:

[0134] Step 206: For the target parking space with 0 observations in the current parking space tracking result, determine the contour point coordinates of the target parking space in the world coordinate system based on the second state quantity of the target parking space obtained from the previous target frame set, and use it as the target position information of the target parking space.

[0135] In this optimization process, parking spaces with zero observations indicate that they slid out of the window and completed optimization upon entering the current process. Therefore, the second state variable obtained from the previous target frame set is used as the final optimized state variable. Based on this, the contour point coordinates of the parking space in the world coordinate system are determined, and these contour point coordinates are used as the target location information for the parking space. The contour point coordinates can include the coordinates of the four corner points of the parking space. The optimized second state variable includes the coordinates of the two endpoints of the entry line, which are also the coordinates of the two corner points of the parking space. Based on these two corner point coordinates and other relevant information about the parking space (such as the parking space type, the direction of the long side of the parking space, the general length of the parking space, the width, etc.), the coordinates of the other two corner points can be determined, thus obtaining the contour point coordinates of the parking space as the target location information.

[0136] In an optional example, step 201, determining the current parking space observation information in the vehicle coordinate system corresponding to the current target frame set, includes:

[0137] Step 2011: Determine the bird's-eye view image in the vehicle coordinate system corresponding to the current frame in the current target frame set.

[0138] The bird's-eye view image in the vehicle coordinate system corresponding to the current frame can be obtained by transforming the current frame image captured by the camera through IPM (Inverse Perspective Mapping). The specific principle will not be elaborated here.

[0139] Step 2012: Based on the bird's-eye view image, determine the current frame parking space observation information in the vehicle coordinate system corresponding to the current frame.

[0140] The parking space observation information of the current frame can be obtained by sensing the bird's-eye view image based on a preset perception algorithm or model, thereby obtaining the parking space information contained therein. The specific principle will not be elaborated here.

[0141] Step 2013: Determine the current parking space observation information based on the current frame parking space observation information and the pre-obtained preset number of historical frame parking space observation information.

[0142] Specifically, since parking space observation is performed for each new frame and the corresponding parking space observation information can be stored after observation, under the current target frame set, it is only necessary to observe the current frame to obtain the current frame parking space observation information, and then obtain the historical frame parking space observation information from the storage area to obtain the current parking space observation information corresponding to the current target frame set.

[0143] Similarly, when determining the current parking space tracking result based on the current parking space observation information, the current parking space tracking result can also be determined by combining the historical parking space tracking results. The specific principle will not be elaborated here.

[0144] In an optional example, step 202, which determines the current parking space tracking result based on the current parking space observation information, includes:

[0145] Step 2021: Based on the current parking space observation information, determine the intersection-over-union ratio (IoU) of each parking space in the current frame with that of each parking space in the previous frame.

[0146] Intersection over Union (IOU) is the ratio of the intersection to the union of parking spaces in two frames. Since the time between two frames is very short, the same parking space will not change much in adjacent frames. Based on the IOU, the correspondence between parking spaces in two frames can be determined, thereby realizing parking space tracking.

[0147] Step 2022: Determine the tracking result of the current parking space based on the intersection-over-union ratio of each parking space in the current frame with that of each parking space in the previous frame.

[0148] Specifically, based on the intersection-over-union ratio (IoU) of each parking space, the correspondence between the parking spaces in the current frame and those in the previous frame can be determined. The parking spaces in the previous frame have already been tracked and have corresponding parking space IDs. Based on the correspondence between the parking spaces in the current frame and those in the previous frame, the parking space IDs of each parking space in the current frame can be determined. For newly observed parking spaces in the current frame, parking space IDs can be set according to the parking space ID setting rules for tracking in subsequent frames. The specific principle of parking space tracking will not be elaborated here.

[0149] In an optional example, step 203, which determines the first state quantity corresponding to each parking space in the current target frame set based on the current parking space tracking result, includes: for parking spaces that already exist in the parking space tracking result corresponding to the previous target frame set, using the second state quantity of the parking space determined based on the previous target frame set as the first state quantity of the parking space in the current target frame set; and for parking spaces newly added in the current parking space tracking result, determining the first state quantity of the parking space based on the parking space observation information corresponding to the parking space in the current parking space observation information.

[0150] Among them, parking space observation information is parking space information in the vehicle coordinate system, which can determine the center point, direction, and length of the entry line in the vehicle coordinate system. Through the transformation from the vehicle coordinate system to the world coordinate system, the first state variable in the world coordinate system is obtained. Details will not be elaborated further.

[0151] This disclosure reconstructs parking spaces based on IPM (in-the-moment) images. It performs 3D reconstruction only on each vertex of the parking space outline in the IPM image, eliminating the need to first reconstruct a point cloud on the ground using LiDAR and then extract parking space parameters from that point cloud. This effectively reduces parking space reconstruction costs. Furthermore, by minimizing the reprojection error of each vertex and constructing adjacency constraints for the parking space, this disclosure obtains parking space outline point coordinates that are more physically accurate. The IPM image can be obtained using a mass-produced fisheye camera, further reducing parking space reconstruction costs.

[0152] Any of the parking space reconstruction methods provided in this disclosure can be executed by any suitable device with data processing capabilities, including but not limited to: terminal devices and servers. Alternatively, any of the parking space reconstruction methods provided in this disclosure can be executed by a processor, such as by a processor executing any of the parking space reconstruction methods mentioned in this disclosure by calling corresponding instructions stored in memory. Further details will not be elaborated below.

[0153] Exemplary device

[0154] Figure 8 This is a schematic diagram of a parking space reconstruction device provided in an exemplary embodiment of this disclosure. The device in this embodiment can be used to implement corresponding method embodiments of this disclosure, such as… Figure 8 The device shown includes: a first determining module 501, a first processing module 502, a second determining module 503, a second processing module 504, and a third processing module 505.

[0155] The first determining module 501 is used to determine the current parking space observation information in the vehicle coordinate system corresponding to the current target frame set, the current target frame set including the current frame and a preset number of historical frames; the first processing module 502 is used to determine the current parking space tracking result based on the current parking space observation information determined by the first determining module 501; the second determining module 503 is used to determine the first state quantity corresponding to each parking space in the current target frame set based on the current parking space tracking result determined by the first processing module 502, the first state quantity including the parking space information of the corresponding parking space in the world coordinate system; the second processing module 504 is used to optimize each first state quantity based on the current parking space observation information, the current parking space tracking result and the preset optimization rules to obtain the optimized second state quantity corresponding to each parking space in the current target frame set; the third processing module 505 is used to determine the target position information in the world coordinate system corresponding to each parking space based on the second state quantity corresponding to each parking space in the current target frame set obtained by the second processing module 504.

[0156] In one optional example, Figure 9This is a schematic diagram of the structure of the second processing module 504 provided in an exemplary embodiment of the present disclosure. In this example, the second processing module 504 includes: a first processing unit 5041, a second processing unit 5042, and a third processing unit 5043.

[0157] The first processing unit 5041 is used to determine the first endpoint coordinates and the second endpoint coordinates of the parking line corresponding to each parking space in the current target frame set in the vehicle coordinate system based on each first state variable and the vehicle pose corresponding to each frame in the current target frame set; the second processing unit 5042 is used to determine the first observation endpoint coordinates and the second observation endpoint coordinates of the parking line corresponding to each parking space observed in the current target frame set based on the current parking space observation information; the third processing unit 5043 is used to optimize each first state variable based on the first endpoint coordinates, second endpoint coordinates, first observation endpoint coordinates, second observation endpoint coordinates, and preset objective function corresponding to each parking space in the current target frame set to obtain each second state variable.

[0158] In one optional example, Figure 10 This is a schematic diagram of the structure of the first processing unit 5041 provided in an exemplary embodiment of this disclosure. In this example, the first state quantity includes the coordinates of the center point of the parking space entry line in the world coordinate system, the heading angle of the entry line, and the length of the entry line; the first processing unit 5041 includes: a first processing subunit 50411 and a second processing subunit 50412.

[0159] The first processing subunit 50411 is used to determine the coordinates of the third endpoint and the fourth endpoint of the entry line corresponding to each parking space in the current target frame set in the world coordinate system based on the length of the entry line, the heading angle of the entry line and the center point coordinates of the entry line in each first state variable; the second processing subunit 50412 is used to determine the coordinates of the first endpoint and the second endpoint of each parking space in the vehicle coordinate system of its corresponding frame based on the coordinates of the third endpoint and the fourth endpoint of each parking space and the vehicle pose corresponding to each frame in the current target frame set.

[0160] In one optional example, Figure 11 This is a schematic diagram of the structure of a third processing unit 5043 provided in an exemplary embodiment of the present disclosure. In this example, the third processing unit 5043 includes: a third processing subunit 50431, a fourth processing subunit 50432, and a fifth processing subunit 50433.

[0161] The third processing subunit 50431 is used to determine the target function value based on the first endpoint coordinates, second endpoint coordinates, first observation endpoint coordinates, and second observation endpoint coordinates corresponding to each parking space in the current target frame set, as well as a preset target function. The fourth processing subunit 50432 is used to update each first state variable based on the target function value using a least squares algorithm to obtain the third state variable corresponding to each first state variable. The fifth processing subunit 50433, in response to each third state variable not meeting the preset conditions, uses each third state variable as a first state variable and optimizes it again, and so on, until each third state variable meets the preset conditions, and then uses each third state variable as a second state variable.

[0162] In an optional example, the second processing unit 5042 is further configured to determine, based on the current parking space observation information, the coordinates of the observation center point, the coordinates of the first observation endpoint, and the coordinates of the second observation endpoint corresponding to each parking space observed in the current target frame set; the third processing subunit 50431 is specifically configured to: determine the endpoint reprojection residual function value in the preset objective function based on the first endpoint coordinates, the second endpoint coordinates, the first observation endpoint coordinates, and the second observation endpoint coordinates corresponding to each parking space in the current target frame set; determine the adjacent parking space constraint residual function value in the preset objective function based on the observation center point coordinates corresponding to each parking space in the current target frame set and the length of the parking line in the first state variable corresponding to each parking space; and determine the objective function value based on the endpoint reprojection residual function value and the adjacent parking space constraint residual function value.

[0163] Figure 12 This is a schematic diagram of the structure of a parking space reconstruction device provided in another exemplary embodiment of this disclosure.

[0164] In an optional example, the apparatus of this disclosure further includes: a fourth processing module 506, configured to determine adjacent parking space groups in the current target frame set based on current parking space observation information, current parking space tracking results, and preset grouping rules; correspondingly, the second processing module 504 includes: a fourth processing unit 5041a, configured to optimize each first state variable based on current parking space observation information, current parking space tracking results, adjacent parking space groups, preset adjacent parking space constraint rules, and preset optimization rules, to obtain the second state variables corresponding to each parking space in the current target frame set.

[0165] Figure 13 This is a schematic diagram of the structure of the third processing module 505 provided in an exemplary embodiment of this disclosure.

[0166] In an optional example, the third processing module 505 includes: a first determining unit 5051, a second determining unit 5052, and a third determining unit 5053.

[0167] The first determining unit 5051 is used to determine the next target frame set based on the current target frame set and the next frame; the second determining unit 5052 is used to take the next target frame set as the current target frame set, and in response to the observation count of the target parking space in the parking space tracking result corresponding to the current target frame set being 0, determine the contour point coordinates of the target parking space in the world coordinate system based on the second state variable corresponding to the target parking space, as the target position information of the target parking space; the third determining unit 5053 is used to take the second state variable corresponding to the parking space as the first state variable for parking spaces with an observation count greater than 0 in the parking space tracking result corresponding to the current target frame set, and continue to optimize until the observation count of the parking space is 0, thereby obtaining the target position information of the parking space.

[0168] Figure 14 This is a schematic diagram of the structure of a parking space reconstruction device provided in another exemplary embodiment of this disclosure.

[0169] In an optional example, the current parking space tracking result includes the number of observations for each tracked parking space; the apparatus of this disclosure further includes: a fifth processing module 507, used to determine the contour point coordinates of the target parking space in the world coordinate system based on the second state quantity of the target parking space obtained from the previous target frame set for the target parking space with 0 observations in the current parking space tracking result, as the target position information of the target parking space.

[0170] In one optional example, Figure 15 This is a schematic diagram of the structure of a first determining module 501 provided in an exemplary embodiment of the present disclosure. In this example, the first determining module 501 includes: a fourth determining unit 5011, a fifth determining unit 5012, and a sixth determining unit 5013.

[0171] The fourth determining unit 5011 is used to determine the bird's-eye view image in the vehicle coordinate system corresponding to the current frame in the current target frame set; the fifth determining unit 5012 is used to determine the parking space observation information of the current frame in the vehicle coordinate system corresponding to the current frame based on the bird's-eye view image; the sixth determining unit 5013 is used to determine the current parking space observation information based on the current frame parking space observation information and a preset number of historical frame parking space observation information obtained in advance.

[0172] In one optional example, Figure 16 This is a schematic diagram of the structure of a first processing module 502 provided in an exemplary embodiment of the present disclosure. In this example, the first processing module 502 includes a seventh determining unit 5021 and an eighth determining unit 5022.

[0173] The seventh determining unit 5021 is used to determine the intersection-over-union ratio (IoU) of each parking space in the current frame with that of each parking space in the previous frame based on the current parking space observation information; the eighth determining unit 5022 is used to determine the current parking space tracking result based on the IoU of each parking space in the current frame with that of each parking space in the previous frame.

[0174] In one optional example, Figure 17 This is a schematic diagram of the structure of the second determining module 503 provided in an exemplary embodiment of the present disclosure. In this example, the second determining module 503 includes a fifth processing unit 5031 and a sixth processing unit 5032.

[0175] The fifth processing unit 5031 is used to, for a parking space that already exists in the parking space tracking result corresponding to the previous target frame set, use the second state quantity of the parking space determined based on the previous target frame set as the first state quantity of the parking space in the current target frame set; the sixth processing unit 5032 is used to, for a parking space newly added in the current parking space tracking result, determine the first state quantity of the parking space based on the parking space observation information corresponding to the parking space in the current parking space observation information.

[0176] The various embodiments or optional examples of the apparatus disclosed herein can be implemented individually or in combination in any way without conflict.

[0177] Exemplary electronic devices

[0178] This disclosure also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program stored in the memory, wherein when the computer program is executed, it implements the parking space reconstruction method described in any of the above embodiments of this disclosure.

[0179] Figure 18 This is a schematic diagram of an application embodiment of the electronic device disclosed herein. In this embodiment, the electronic device 10 includes one or more processors 11 and a memory 12.

[0180] The processor 11 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 10 to perform desired functions.

[0181] The memory 12 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 11 may execute the program instructions to implement the methods of the various embodiments of this disclosure described above and / or other desired functions. Various contents such as input signals, signal components, and noise components may also be stored in the computer-readable storage medium.

[0182] In one example, the electronic device 10 may also include an input device 13 and an output device 14, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0183] For example, the input device 13 may be the microphone or microphone array described above, used to capture the input signal of the sound source.

[0184] In addition, the input device 13 may also include, for example, a keyboard, a mouse, etc.

[0185] The output device 14 can output various information to the outside, including determined distance information, direction information, etc. The output device 14 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0186] Of course, for the sake of simplicity, Figure 18 Only some of the components of the electronic device 10 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 10 may include any other suitable components depending on the specific application.

[0187] Exemplary computer program products and computer-readable storage media

[0188] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps of the methods according to various embodiments of this disclosure as described in the "Exemplary Methods" section of this specification.

[0189] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0190] Furthermore, embodiments of this disclosure may also be computer-readable storage media having computer program instructions stored thereon, which, when executed by a processor, cause the processor to perform the steps in the methods according to various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0191] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0192] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0193] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0194] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0195] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0196] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0197] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0198] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A parking space reconstruction method, comprising: Determine the current parking space observation information in the vehicle coordinate system corresponding to the current target frame set, wherein the current target frame set includes the current frame and a preset number of historical frames; The current parking space observation information includes parking space-related information obtained from observations of each frame in the current target frame set; Based on the current parking space observation information, determine the current parking space tracking result; Based on the current parking space tracking result, a first state quantity corresponding to each parking space in the current target frame set is determined. The first state quantity includes the parking space information of the corresponding parking space in the world coordinate system. Based on the current parking space observation information, the current parking space tracking results, and the preset optimization rules, each of the first state variables is optimized to obtain the optimized second state variables corresponding to each parking space in the current target frame set. The preset optimization rules include an objective function determined based on adjacent parking space constraints and parking space entry line endpoint observation reprojection constraints. The optimized second state variables corresponding to each parking space are obtained by minimizing the objective function value of each parking space under the constraint conditions. Based on the second state quantity corresponding to each parking space in the current target frame set, the target position information in the world coordinate system corresponding to each parking space is determined.

2. The method according to claim 1, wherein, The step of optimizing each of the first state variables based on the current parking space observation information, the current parking space tracking result, and preset optimization rules to obtain the optimized second state variables corresponding to each parking space in the current target frame set includes: Based on each of the first state variables and the vehicle pose corresponding to each frame in the current target frame set, the coordinates of the first endpoint and the second endpoint of the entry line corresponding to each parking space in the current target frame set in the vehicle coordinate system are determined. Based on the current parking space observation information, determine the coordinates of the first and second observation endpoints of the entry line corresponding to each parking space observed in the current target frame set. Based on the first endpoint coordinates, second endpoint coordinates, first observation endpoint coordinates, and second observation endpoint coordinates corresponding to each parking space in the current target frame set, and a preset objective function, each of the first state variables is optimized to obtain each of the second state variables.

3. The method according to claim 2, wherein, The first state quantity includes the coordinates of the center point of the corresponding parking space's entry line in the world coordinate system, the heading angle of the entry line, and the length of the entry line. The step of determining the coordinates of the first endpoint and the second endpoint of the entry line corresponding to each parking space in the current target frame set in the vehicle coordinate system based on each of the first state variables and the vehicle poses corresponding to each frame in the current target frame set includes: Based on the length of the entry line, the heading angle of the entry line, and the center point coordinates of the entry line in each of the first state variables, the third endpoint coordinates and the fourth endpoint coordinates of the entry line corresponding to each parking space in the current target frame set in the world coordinate system are determined respectively. Based on the third endpoint coordinates and the fourth endpoint coordinates corresponding to each parking space, and the vehicle pose corresponding to each frame in the current target frame set, the first endpoint coordinates and the second endpoint coordinates corresponding to each parking space in the vehicle coordinate system of its corresponding frame are determined.

4. The method according to claim 2, wherein, The optimization of each first state variable based on the first endpoint coordinates, second endpoint coordinates, first observation endpoint coordinates, and second observation endpoint coordinates corresponding to each parking space in the current target frame set, and a preset objective function, to obtain each second state variable, includes: Based on the first endpoint coordinates, second endpoint coordinates, first observation endpoint coordinates, and second observation endpoint coordinates corresponding to each parking space in the current target frame set, and a preset target function, the target function value is determined. Based on the objective function value, the least squares algorithm is used to update each of the first state variables to obtain the third state variables corresponding to each of the first state variables. In response to each of the third state quantities not meeting the preset conditions, each of the third state quantities is used as each of the first state quantities, and optimization is performed again, and so on, until each of the third state quantities meets the preset conditions, and then each of the third state quantities is used as each of the second state quantities.

5. The method according to claim 4, wherein, The step of determining the coordinates of the first and second observation endpoints of the entry line corresponding to each parking space observed in the current target frame set, based on the current parking space observation information, includes: Based on the current parking space observation information, determine the coordinates of the observation center point, the first observation endpoint, and the second observation endpoint of the parking space corresponding to each parking space observed in the current target frame; The step of determining the target function value based on the first endpoint coordinates, second endpoint coordinates, first observation endpoint coordinates, and second observation endpoint coordinates corresponding to each parking space in the current target frame set, and a preset target function, includes: Based on the first endpoint coordinates, second endpoint coordinates, first observation endpoint coordinates, and second observation endpoint coordinates corresponding to each parking space in the current target frame set, the endpoint reprojection residual function value in the preset objective function is determined; Based on the coordinates of the observation center point corresponding to each parking space in the current target frame set, and the length of the entry line in the first state variable corresponding to each parking space, the value of the adjacent parking space constraint residual function in the preset objective function is determined. The objective function value is determined based on the endpoint reprojection residual function value and the adjacent parking space constraint residual function value.

6. The method according to claim 1, wherein, After determining the current parking space tracking result based on the current parking space observation information, the method further includes: Based on the current parking space observation information, the current parking space tracking result, and the preset grouping rules, the adjacent parking spaces in the current target frame set are grouped. The step of optimizing each of the first state variables based on the current parking space observation information, the current parking space tracking result, and preset optimization rules to obtain the optimized second state variables corresponding to each parking space in the current target frame set includes: Based on the current parking space observation information, the current parking space tracking result, the adjacent parking space grouping, the preset adjacent parking space constraint rules and the preset optimization rules, each of the first state variables is optimized to obtain the second state variables corresponding to each parking space in the current target frame set.

7. The method according to claim 1, wherein, The step of determining the target position information in the world coordinate system corresponding to each parking space based on the second state quantity corresponding to each parking space in the current target frame set includes: Based on the current target frame set and the next frame, determine the next target frame set; The next target frame set is taken as the current target frame set. In response to the observation count of the target parking space in the parking space tracking result corresponding to the current target frame set being 0, the contour point coordinates of the target parking space in the world coordinate system are determined based on the second state variable corresponding to the target parking space, and are used as the target position information of the target parking space. For parking spaces in the current target frame set whose observation count is greater than 0 in the parking space tracking results, the second state variable corresponding to the parking space is used as the first state variable, and optimization is continued until the observation count of the parking space is 0, thereby obtaining the target position information of the parking space.

8. The method according to claim 1, wherein, The current parking space tracking result includes the number of observations for each tracked parking space; After determining the current parking space tracking result based on the current parking space observation information, the method further includes: For a target parking space with 0 observations in the current parking space tracking result, the contour point coordinates of the target parking space in the world coordinate system are determined based on the second state variable of the target parking space obtained from the previous target frame set, and are used as the target position information of the target parking space.

9. The method according to claim 1, wherein, The determination of the current parking space observation information in the vehicle coordinate system corresponding to the current target frame set includes: Determine the bird's-eye view image in the vehicle coordinate system corresponding to the current frame in the current target frame set; Based on the bird's-eye view image, determine the parking space observation information of the current frame in the vehicle coordinate system corresponding to the current frame; The current parking space observation information is determined based on the current frame parking space observation information and the preset number of historical frame parking space observation information obtained in advance.

10. The method according to any one of claims 1-9, wherein, The step of determining the current parking space tracking result based on the current parking space observation information includes: Based on the current parking space observation information, determine the intersection-over-union ratio (IoU) of each parking space in the current frame with each parking space in the previous frame; The tracking result of the current parking space is determined based on the intersection-over-union ratio (IoU) of each parking space in the current frame with that of each parking space in the previous frame.

11. The method according to any one of claims 1-9, wherein, The step of determining the first state variable corresponding to each parking space in the current target frame set based on the current parking space tracking result includes: For a parking space that already exists in the parking space tracking result corresponding to the previous target frame set, the second state quantity of the parking space determined based on the previous target frame set is used as the first state quantity of the parking space in the current target frame set. For a newly added parking space in the current parking space tracking results, the first state variable of the parking space is determined based on the parking space observation information corresponding to the parking space in the current parking space observation information.

12. A parking space reconstruction device, comprising: The first determining module is used to determine the current parking space observation information in the vehicle coordinate system corresponding to the current target frame set, wherein the current target frame set includes the current frame and a preset number of historical frames; The current parking space observation information includes parking space-related information obtained from observations of each frame in the current target frame set; The first processing module is used to determine the current parking space tracking result based on the current parking space observation information; The second determining module is used to determine the first state quantity corresponding to each parking space in the current target frame set based on the current parking space tracking result. The first state quantity includes the parking space information of the corresponding parking space in the world coordinate system. The second processing module is used to optimize each of the first state variables based on the current parking space observation information, the current parking space tracking result, and preset optimization rules to obtain the optimized second state variables corresponding to each parking space in the current target frame set. The preset optimization rules include an objective function determined based on adjacent parking space constraints and parking space entry line endpoint observation reprojection constraints. The optimized second state variables corresponding to each parking space are obtained by minimizing the objective function value of each parking space under the constraint conditions. The third processing module is used to determine the target position information in the world coordinate system corresponding to each parking space based on the second state quantity corresponding to each parking space in the current target frame set.

13. A computer-readable storage medium storing a computer program for performing the parking space reconstruction method according to any one of claims 1-11.

14. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the parking space reconstruction method according to any one of claims 1-11.

Images