3D semantic occupation prediction method for parking space occupation detection of parking lot

Abstract

The invention discloses a 3D semantic occupancy prediction method for parking space occupancy detection of a parking lot, and belongs to the technical field of vehicle parking detection. The method comprises the following steps: collecting image data of a parking lot scene through a camera in a parking lot; extracting image features from the collected image data, and predicting depth information of each pixel point in the image; reconstructing a 3D occupation grid of the parking lot scene according to the extracted image features and the predicted depth information; performing accurate semantic classification / identification on each voxel in a space in a 3D occupied grid under depth guidance, and reconstructing a complex three-dimensional scene with a semantic label, including an occlusion area and an image invisible area; and carrying out parking space occupation prediction based on the reconstructed three-dimensional scene, and identifying a parking space abnormal condition according to an occupation prediction result. According to the invention, 3D reconstruction is carried out by combining the depth information and the image features, and the problem of insufficient precision in a 3D reconstruction method based on monocular vision can be effectively solved.

Description

Technical Field

[0001] The present invention belongs to the technical field of vehicle parking detection, and in particular relates to a 3D semantic occupancy prediction method for parking lot parking space occupancy detection. Background Art

[0002] Traditional parking lot management systems mainly rely on cameras for monitoring, but due to the limitation of two-dimensional image information, it is difficult to accurately judge the parking situation of vehicles. For example, in the case of vehicle occlusion, light changes, etc., the recognition accuracy is low, which is easy to cause false alarms or missed alarms, affecting the management efficiency and safety of parking lots.

[0003] In order to solve the above problems, researchers have proposed a variety of vehicle detection and recognition methods based on deep learning. For example, target detection algorithms such as YOLO, SSD, and Faster R-CNN can identify vehicles in images, but it is difficult to obtain the precise location and size information of the vehicle. Point cloud processing algorithms such as PointNet and PointNet++ can process three-dimensional point cloud data, but it is difficult to obtain the semantic information of the vehicle. Semantic segmentation algorithms based on deep learning can segment semantic categories such as vehicles, ground, and parking spaces in images, but it is difficult to obtain the precise location and size information of the vehicle.

[0004] In addition, the cost of collecting point cloud data is high, and it is difficult to obtain the semantic information of the target. For example, using equipment such as LiDAR to collect point cloud data requires a high cost, and it is difficult to distinguish different categories of target objects, such as vehicles, pedestrians, and the ground.

[0005] Existing deep learning-based vehicle detection and recognition methods, including target detection, point cloud processing, and semantic segmentation, have certain limitations in solving the problem of abnormal vehicle detection in parking lot management. The target detection algorithm is difficult to obtain the precise location and size information of the target, the point cloud processing algorithm has a high acquisition cost and is difficult to obtain the semantic information of the target, and the semantic segmentation algorithm is difficult to obtain the precise location and size information of the target.

[0006] It can be seen that the prior art has the following shortcomings: Existing methods are mainly based on two-dimensional image information, and it is difficult to accurately determine whether the parking position of the vehicle is standardized, such as whether it exceeds the parking space range, whether it is parked in a prohibited parking area, etc. Limited by two-dimensional image information, it is difficult to accurately determine the parking situation of the vehicle, especially in cases of vehicle occlusion, lighting changes, etc., the recognition accuracy is low. It is also difficult to distinguish different types of target objects, such as vehicles, pedestrians, and the ground, and it is difficult to detect abnormal situations such as vehicles parked in prohibited parking areas. Summary of the invention

[0007] The purpose of an embodiment of the present invention is to provide a 3D semantic occupancy prediction method for parking lot parking space occupancy detection, which combines three-dimensional spatial data and deep learning technology to improve the recognition accuracy of vehicle parking positions and reduce the impact of occupancy and lighting changes, thereby effectively solving the problem of parking space occupancy detection in parking lot management, improving the management efficiency and safety of parking lots, and further solving at least one technical problem involved in the background technology.

[0008] In order to solve the above-mentioned technical problems, the present invention is achieved as follows: In a first aspect, an embodiment of the present invention provides a 3D semantic occupancy prediction method for parking lot parking space occupancy detection, comprising the following steps: Step S1, collecting image data of a parking lot scene through a camera in the parking lot; Step S2, extracting image features from the collected image data, and introducing DepthAnything to predict the depth information of each pixel in the image; Step S3, reconstructing the 3D occupied grid space of the parking lot scene based on the extracted image features and the predicted depth information; Step S4, in the 3D occupied grid space under the guidance of depth, accurately semantically classify / recognize each voxel in the space, and reconstruct a complex 3D scene with semantic labels including occluded and image invisible areas; Step S5, performing parking space occupancy prediction based on the reconstructed complex three-dimensional scene, and then identifying abnormal parking space conditions according to the occupancy prediction results.

[0009] In a second aspect, an embodiment of the present invention provides an electronic device, including: at least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the steps of the method described in the first aspect.

[0010] In a third aspect, an embodiment of the present invention provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0011] In a fourth aspect, an embodiment of the present invention provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0012] Compared with the prior art, the present invention has the following beneficial effects: 1. The present invention combines depth information and image features to perform 3D reconstruction, which can effectively solve the problem of insufficient accuracy in traditional 3D reconstruction methods based on monocular vision; 2. Compared with traditional 3D reconstruction methods that often rely on complex equipment such as binocular vision or structured light, which are costly and difficult to deploy, the present invention only requires ordinary cameras, greatly reducing hardware costs and installation complexity; in addition, through accurate 3D reconstruction, the actual scene of the parking lot can be restored more realistically, providing a reliable data basis for subsequent parking space status analysis, thereby improving the accuracy of parking space occupancy detection; 3. The present invention uses depth-guided 3D occupancy grids for semantic segmentation and occupancy prediction, solving the problem that traditional methods are difficult to accurately identify occluded and invisible areas. By mapping 2D image features to 3D space and using the Transformer architecture for feature interaction, the semantic information of the 3D feature map is enhanced, enabling accurate object recognition and classification even in complex and changeable parking environments. 4. Based on the occupancy prediction results, the present invention can automatically identify abnormal parking conditions, such as vehicles parked beyond the range, obstacles in the parking space, etc., so as to promptly issue an alarm or notify the management staff to take corresponding measures. This not only helps to improve the management efficiency and service quality of the parking lot, but also effectively avoids safety hazards caused by improper use of parking spaces; in addition, by setting a reasonable abnormal score threshold, the system can flexibly adjust the sensitivity to abnormal conditions, which not only ensures the accuracy of detection, but also reduces the false alarm rate, ensuring the stable operation of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work, among which: Figure 1 It is a flow chart of a 3D semantic occupancy prediction method for parking lot parking space occupancy detection provided by an embodiment of the present invention; Figure 2 This is one of the hardware structure diagrams of the electronic device provided by the embodiment of the present invention; Figure 3 This is the second schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present invention. DETAILED DESCRIPTION

[0014] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0015] The terms "first", "second", etc. in the specification and claims of the present invention are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0016] The 3D semantic occupancy prediction method for parking lot parking space occupancy detection provided by the embodiment of the present invention is described in detail below with reference to the accompanying drawings through specific embodiments and application scenarios.

[0017] See also Figure 1 , is a 3D semantic occupancy prediction method for parking lot parking space occupancy detection provided by an embodiment of the present invention, comprising the following steps: Step S1, collecting image data of a parking lot scene through a camera in the parking lot; Step S2, extracting image features from the collected image data, and introducing DepthAnything to predict the depth information of each pixel in the image; Step S3, reconstructing the 3D occupied grid space of the parking lot scene based on the extracted image features and the predicted depth information; Step S4, in the 3D occupied grid space under the guidance of depth, accurately semantically classify / recognize each voxel in the space, and reconstruct a complex 3D scene with semantic labels including occluded and image invisible areas; Step S5, performing parking space occupancy prediction based on the reconstructed complex three-dimensional scene, and then identifying abnormal parking space conditions according to the occupancy prediction results.

[0018] In step S1, the image data is a two-dimensional color image I ( u , v ),in, ( u , v ) are the pixel coordinates on the image plane.

[0019] In step S2, ResNet-50 is used to extract image features, including: Image I ( u , v ) Input ResNet-50, output image I ( u , v ) At each pixel coordinate ( u , v ) at the eigenvector F ( u , v ), whose dimensions are B × d × H × W , where B represents the number of images processed in one forward propagation; d represents the number of channels; H represents the height of the output feature map; and W represents the width of the output feature map.

[0020] In step S2, DepthAnything is introduced to predict the depth information of each pixel in the image, including: The feature vector F ( u , v ) Input DepthAnything, output image I ( u , v ) At each pixel coordinate ( u , v ) The predicted depth information ( u , v ), whose dimensions are B ×1× H × W , where 1 indicates the number of channels is 1.

[0021] Step S3 specifically includes: Step S31, using depth information ( u , v ) and the camera intrinsic matrix K The 2D image I ( u , v ) pixel coordinates( u , v ) converted to 3D space coordinates ( X , Y , Z ), which is expressed by the following formula: ; In the formula, is the camera optical center coordinate, and is the focal length in the horizontal and vertical directions; Step S32, using the Transformer architecture to propagate features, transforming the 2D feature vector F ( u , v ) is transferred to the 3D space and a 3D feature map is generated ; Step S33: Use the Transformer architecture to perform feature interaction and enhance the 3D feature map semantic information and generate a complete 3D semantic scene .

[0022] Step S32 specifically includes: The predefined voxel query Q The corresponding 2D feature vector F ( u , v ) Perform cross-attention mechanism calculation to obtain the 3D feature vector of the voxel query, and then generate a 3D feature map , which is expressed by the following formula: ; ; In the formula, is the set of voxel queries Specific voxel query in ; is a 2D image feature sequence; Is a voxel query The corresponding visible image set; P (p,t) is the function that projects the 3D point p onto the image t; is the 2D feature of image t; g is the aggregation function; Is to update the voxel query The subsequent collection is is a set of ; m is a mask tag; DCA is a deformable cross attention mechanism; DA is a deformable attention mechanism.

[0023] Step S33 specifically includes: The 3D feature map The voxel feature vectors are calculated by the self-attention mechanism to enhance the semantic association between feature vectors and aggregate the interacted feature vectors to obtain a complete 3D semantic scene. , which is expressed by the following formula: ; In the formula, DSA is a deformable self-attention mechanism; f is a voxel query or mask token; p is the location of the voxel query or mask token.

[0024] In step S4, each voxel in the space is accurately semantically classified / identified to obtain voxel features , Represents a low-resolution feature map; upsamples and projects it to the output space to obtain the output , where M +1 represents M semantic classes and one empty class, i.e., the result of occupancy prediction; Z represents the depth dimension.

[0025] In step S5, identifying abnormal parking space conditions according to the occupancy prediction results specifically includes: Step S51, calculate the abnormality score, input the voxel feature vector of the 3D feature map , the output 3D voxel anomaly score s(x) is expressed as follows:

[0026] In the formula, the anomaly score s(x) indicates the possibility that the voxel belongs to an abnormal object; g is a nonlinear transformation function; x is the input data; y is the corresponding label; is the unnormalized estimate of the joint probability density; Step S52, threshold processing, compare the abnormal score s(x) with the set threshold τ, if s(x)>τ, then mark the voxel as an abnormal voxel, which is expressed by the following formula: A(x)=s(x)>τ Where A(x) represents the 3D abnormal voxel map, A(x)=1 represents an abnormal voxel, A(x)=0 represents a normal voxel, and τ represents the threshold; Step S53, fusion of 3D semantic scene, 3D abnormal voxel map A(x) and the complete 3D semantic scene After fusion, the abnormal voxels in the fused 3D semantic scene are marked as "unknown", which is expressed as follows:

[0027] In the formula, Represents the fused 3D semantic occupancy prediction result; Represents a complete 3D semantic scene.

[0028] like Figure 2As shown, an embodiment of the present invention further provides an electronic device 600, which includes a processor 601, a memory 602, and a program or instruction stored in the memory 602 and executable on the processor 601. When the program or instruction is executed by the processor 601, each process of the above-mentioned 3D semantic occupancy prediction method embodiment for parking lot parking space occupancy detection is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0029] It should be noted that the electronic devices in the embodiments of the present invention include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0030] Figure 3 The present invention is a schematic diagram of the hardware structure of an electronic device.

[0031] The electronic device 700 includes but is not limited to: a radio frequency unit 701, a network module 702, an audio output unit 703, an input unit 704, a sensor 705, a display unit 706, a user input unit 707, an interface unit 708, a memory 709, and a processor 710.

[0032] Those skilled in the art will appreciate that the electronic device 700 may also include a power source (such as a battery) for supplying power to various components, and the power source may be logically connected to the processor 710 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 3 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0033] It should be understood that in the embodiment of the present invention, the input unit 704 may include a graphics processing unit (GPU) 7041 and a microphone 7042, and the graphics processor 7041 processes the image data of the static image or video obtained by the image capture device (such as a camera) in the video capture mode or the image capture mode. The display unit 706 may include a display panel 7061, and the display panel 7061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 707 includes a touch panel 7071 and other input devices 7072. The touch panel 7071 is also called a touch screen. The touch panel 7071 may include two parts: a touch detection device and a touch controller. Other input devices 7072 may include but are not limited to a physical keyboard, a function key (such as a volume control button, a switch button, etc.), a trackball, a mouse, and a joystick, which will not be repeated here. The memory 709 may be used to store software programs and various data, including but not limited to applications and operating systems. The processor 710 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, the user interface, and the application program, etc., and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 710.

[0034] An embodiment of the present invention also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned 3D semantic occupancy prediction method embodiment for parking lot parking space occupancy detection are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0035] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0036] An embodiment of the present invention further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned 3D semantic occupancy prediction method embodiment for parking space occupancy detection in parking lots, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0037] It should be understood that the chip mentioned in the embodiment of the present invention may also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0038] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the method and device in the embodiment of the present invention is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0039] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0040] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation modes, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the present invention and the claims, all of which are within the protection of the present invention.

Claims

1. A 3D semantic occupancy prediction method for parking lot occupancy detection, characterized in that: The following steps are involved: Step S1, collecting image data of a parking lot scene through a camera in the parking lot; Step S2, extracting image features from the collected image data, and introducing DepthAnything to predict the depth information of each pixel in the image; Step S3, reconstructing the 3D occupied grid space of the parking lot scene based on the extracted image features and the predicted depth information; Step S4, in the 3D occupied grid space under the guidance of depth, accurately semantically classify / recognize each voxel in the space, and reconstruct a complex 3D scene with semantic labels including occluded and image invisible areas; Step S5, performing parking space occupancy prediction based on the reconstructed complex three-dimensional scene, and then identifying abnormal parking space conditions according to the occupancy prediction results.

2. The method according to claim 1, characterized in that In step S1, the image data is a two-dimensional color image I ( u , v ),in,( u , v ) are the pixel coordinates on the image plane.

3. The method according to claim 2, characterized in that In step S2, ResNet-50 is used to extract image features, including: Image I ( u , v ) Input ResNet-50, output image I ( u , v ) At each pixel coordinate ( u , v ) at the eigenvector F ( u , v ), whose dimensions are B × d × H × W , where B represents the number of images processed in one forward propagation; d represents the number of channels; H represents the height of the output feature map; and W represents the width of the output feature map.

4. The method according to claim 3, characterized in that In step S2, DepthAnything is introduced to predict the depth information of each pixel in the image, including: The feature vector F ( u , v ) Input DepthAnything, output image I ( u , v ) At each pixel coordinate ( u , v ) The predicted depth information ( u , v ), whose dimensions are B ×1× H × W , where 1 indicates the number of channels is 1.

5. The method according to claim 4, characterized in that Step S3 specifically includes: Step S31, using depth information ( u , v ) and the camera intrinsic matrix K The 2D image I ( u , v ) pixel coordinates( u , v ) converted to 3D space coordinates ( X , Y , Z ), which is expressed by the following formula: ； In the formula, is the camera optical center coordinate, and is the focal length in the horizontal and vertical directions; Step S32, using the Transformer architecture to propagate features, transforming the 2D feature vector F ( u , v ) is transferred to the 3D space and a 3D feature map is generated ; Step S33: Use the Transformer architecture to perform feature interaction and enhance the 3D feature map semantic information and generate a complete 3D semantic scene .

6. The method according to claim 5, characterized in that Step S32 specifically includes: The predefined voxel query The corresponding 2D feature vector F ( u , v ) Perform cross-attention calculations to obtain the 3D voxel feature vector of the voxel query, and then generate a 3D feature map , expressed by the following formula: ; ； In the formula, is the set of voxel queries Specific voxel query in ; is a 2D image feature sequence; Is a voxel query The corresponding visible image set; P (p,t) is the function that projects the 3D point p onto the image t; is the 2D feature of image t; g is the aggregation function; Is to update the voxel query The subsequent collection is is a set of ; m is a mask tag; DCA is a deformable cross attention mechanism; DA is a deformable attention mechanism.

7. The method according to claim 6, characterized in that Step S33 specifically includes: The 3D feature map The voxel feature vectors are calculated by the self-attention mechanism to enhance the semantic association between feature vectors and aggregate the interacted feature vectors to obtain a complete 3D semantic scene. , expressed by the following formula: ; In the formula, DSA is a deformable self-attention mechanism; f is a voxel query or mask token; p is the location of the voxel query or mask token.

8. The method according to claim 7, characterized in that In step S4, each voxel in the space is accurately semantically classified / identified to obtain voxel features , upsampled and projected to the output space, the output is , where M +1 represents M semantic classes and one empty class, i.e., the result of occupancy prediction; Z represents the depth dimension.

9. The method according to claim 7, characterized in that: In step S5, identifying abnormal parking space conditions according to the occupancy prediction results specifically includes: Step S51, calculate the abnormality score, input the voxel feature vector of the 3D feature map , the output 3D voxel anomaly score s(x) is expressed as follows: ; In the formula, the anomaly score s(x) indicates the possibility that the voxel belongs to an abnormal object; g is a nonlinear transformation function; x is the input data; y is the corresponding label; is the unnormalized estimate of the joint probability density; Step S52, threshold processing, compare the abnormal score s(x) with the set threshold τ, if s(x)>τ, then mark the voxel as an abnormal voxel, which is expressed by the following formula: A(x)=s(x)>τ Where A(x) represents the 3D abnormal voxel map, A(x)=1 represents an abnormal voxel, A(x)=0 represents a normal voxel, and τ represents the threshold; Step S53, fusion of 3D semantic scene, 3D abnormal voxel map A(x) and the complete 3D semantic scene After fusion, abnormal voxels in the fused 3D semantic scene are marked as "unknown", which is expressed as follows: ; In the formula, Represents the fused 3D semantic occupancy prediction result; Represents a complete 3D semantic scene.

Images