Gate machine state determination method, device, equipment, medium and program product

Abstract

This application provides a method, apparatus, device, medium, and program product for determining the status of a turnstile. The method includes: if the distance between a vehicle and the turnstile is less than or equal to a distance threshold, acquiring an image data set through the vehicle's surround-view system; wherein the surround-view system includes at least two image acquisition devices; the image data in the image data set includes at least the features of the turnstile; acquiring radar point cloud data through the vehicle's lidar; the radar point cloud data includes at least the features of the turnstile; and determining the opening / closing status of the turnstile based on the image data set and the radar point cloud data. This application improves the accuracy and stability of determining the opening / closing status of turnstiles, thereby meeting the practical needs of detecting the status of vehicle turnstiles.

Description

Technical Field

[0001] This application relates to gate status detection and processing technology, and more particularly to a gate status determination method, device, equipment, medium, and program product. Background Technology

[0002] In practical applications, vehicles typically collect gate status data using a single type of sensor and determine the gate's open / closed status based on this data. However, in harsh environments, such as those with insufficient lighting, the accuracy of the open / closed status determined in this way is insufficient and cannot meet the actual detection needs of vehicles for gate status. Summary of the Invention

[0003] This application provides a method, apparatus, device, medium, and program product for determining the status of a turnstile, which can improve the accuracy and stability of determining the opening and closing status of a turnstile, thereby meeting the actual needs of turnstile status detection for vehicles.

[0004] The technical solution provided in this application is as follows:

[0005] This application first provides a method for determining the status of a turnstile, the method comprising:

[0006] If the distance between the vehicle and the gate is less than or equal to a distance threshold, an image data set is acquired through the vehicle's surround view system; wherein, the surround view system includes at least two image acquisition devices; the image data set includes at least the features of the gate;

[0007] Radar point cloud data is collected by the vehicle's lidar; wherein the radar point cloud data includes at least the features of the gate.

[0008] Based on the image data set and the radar point cloud data, the opening and closing status of the gate is determined.

[0009] In some embodiments, determining the opening / closing state of the gate based on the image data set and the radar point cloud data includes:

[0010] Feature extraction is performed on the image data in the image dataset to obtain the first image feature;

[0011] Feature extraction is performed on the radar point cloud data to obtain point cloud features;

[0012] Three-dimensional target detection is performed based on the first image features and the point cloud features to obtain the body posture of the gate body; wherein, the gate includes a gate control device and the gate body; the gate control device is used to control the body posture of the gate body to intercept or allow the vehicle to pass.

[0013] Based on the first image features and the point cloud features, occupancy grid detection is performed to obtain the relative positional relationship between the gate body and the vehicle;

[0014] The switch state is determined based on the body's posture and the relative positional relationship.

[0015] In some embodiments, determining the switch state based on the body posture and the relative positional relationship includes:

[0016] If the body posture indicates that the gate body is in a raised state, and the relative position relationship indicates that the gate body is not in a state of blocking the vehicle, then the gate is determined to be in a vehicle-allowing state.

[0017] In some embodiments, the step of performing 3D target detection based on the first image features and the point cloud features to obtain the body pose of the gate body includes:

[0018] The first image features and the point cloud features are fused to obtain the first fusion result;

[0019] The first fusion result is subjected to feature extraction processing to obtain a first processing result; wherein, the first processing result includes at least the ontology pose and confidence level;

[0020] The method further includes:

[0021] The gate body is confirmed based on the confidence level.

[0022] In some embodiments, performing occupancy grid detection based on the first image features and the point cloud features to obtain the relative positional relationship between the gate body and the vehicle includes:

[0023] The first image features and the point cloud features are fused to obtain the first fusion result;

[0024] The first fusion result is processed by an occupancy grid algorithm to obtain a second processing result; wherein, the second processing result includes at least the type and occupancy status of the voxel corresponding to the gate body;

[0025] The relative positional relationship is determined based on the type and occupancy status in the second processing result.

[0026] In some embodiments, fusing the first image features and the point cloud features to obtain a first fusion result includes:

[0027] The first image features and the point cloud features are fused together in a bird's-eye view to obtain a third processing result;

[0028] The first fusion result is obtained by processing the third processing result through an encoder.

[0029] In some embodiments, the method further includes:

[0030] If the radar point cloud data is not obtained, feature extraction processing is performed on the image data set to obtain the second image features;

[0031] Acquire text data; wherein the text data includes at least data for detecting the status of the gate;

[0032] By fusing the text features corresponding to the text data with the second image features, a second fusion result is obtained;

[0033] The second fusion result is processed using a visual language model to determine the opening and closing status of the gate.

[0034] This application embodiment also provides a gate state determination device, the gate state determination device comprising:

[0035] The acquisition module is used to acquire an image data set through the vehicle's surround view system and to acquire radar point cloud data through the vehicle's lidar if the distance between the vehicle and the gate is less than or equal to a distance threshold. The surround view system includes at least two image acquisition devices. The image data set includes at least the features of the gate. The radar point cloud data includes at least the features of the gate.

[0036] The determination module is used to determine the opening and closing status of the gate based on the image data set and the radar point cloud data.

[0037] In some embodiments, the determining module is further configured to extract features from the image data in the image dataset to obtain first image features; extract features from the radar point cloud data to obtain point cloud features; and perform three-dimensional target detection based on the first image features and the point cloud features to obtain the body posture of the gate body; wherein the gate includes a gate control device and the gate body; the gate control device is configured to control the body posture of the gate body to intercept or allow the vehicle to pass.

[0038] The determining module is further configured to perform occupancy grid detection based on the first image features and the point cloud features to obtain the relative positional relationship between the gate body and the vehicle; and to determine the switch state based on the body posture and the relative positional relationship.

[0039] In some embodiments, the determining module is further configured to determine that the gate is in a vehicle-allowing state if the body posture indicates that the gate body is in a raised state and the relative positional relationship indicates that the gate body is not in a state of blocking the vehicle.

[0040] In some embodiments, the determining module is further configured to fuse the first image features and the point cloud features to obtain a first fusion result; perform feature extraction processing on the first fusion result to obtain a first processing result; wherein the first processing result includes at least the ontology pose and confidence level;

[0041] The determining module is also used to confirm the gate body based on the confidence level.

[0042] In some embodiments, the determining module is further configured to fuse the first image features and the point cloud features to obtain a first fusion result; and process the first fusion result using an occupancy grid algorithm to obtain a second processing result; wherein the second processing result includes at least the type and occupancy status of the voxel corresponding to the gate body;

[0043] The determining module is further configured to determine the relative positional relationship based on the type and occupancy status in the second processing result.

[0044] In some embodiments, the determining module is further configured to perform fusion processing on the first image features and the point cloud features in a bird's-eye view space to obtain a third processing result; and process the third processing result by an encoder to obtain the first fusion result.

[0045] In some embodiments, the determining module is further configured to, if the radar point cloud data is not obtained, perform feature extraction processing on the image data set to obtain second image features; and acquire text data; wherein the text data includes at least data for detecting the state of the gate;

[0046] The determining module is further configured to fuse the text features corresponding to the text data with the second image features to obtain a second fusion result; and to process the second fusion result through a visual language model to determine the opening and closing state of the gate.

[0047] This application also provides an electronic device, which includes a processor and a memory; the memory stores a computer program; when the computer program is executed by the processor, it can implement the gate state determination method as described above.

[0048] This application also provides a computer-readable storage medium storing a computer program; when the computer program is executed by a processor of an electronic device, it can implement the gate state determination method as described above.

[0049] This application also provides a computer program product, which includes a computer program; when the computer program is executed by the processor of an electronic device, it can implement the gate state determination method as described above.

[0050] The embodiments of this application have the following beneficial effects:

[0051] The gate state determination method provided in this application embodiment, since the surround view system includes at least two image acquisition devices, when the distance between the vehicle and the gate is less than or equal to a distance threshold, utilizes the image data set acquired by the vehicle's surround view system. Since the image data set includes features of the gate, this not only controls the image data acquisition operation of the surround view system but also allows for a comprehensive and high-precision characterization of the gate's position and attitude relative to the vehicle. Furthermore, the radar point cloud data acquired by the vehicle's lidar, which also includes features of the gate, enables a three-dimensional spatial... It accurately represents the spatial distance and position of the gate relative to the vehicle. Based on this, and using image data sets and radar point cloud data, it determines the gate's on / off state. This allows the gate's on / off state to be associated with the image data sets collected by the surround view system and the radar point cloud data collected by the lidar. It also achieves the fusion of data collected by the surround view system and lidar respectively, which can improve the comprehensiveness and accuracy of the vehicle's perception of the gate and its surrounding environment. This can improve the accuracy of the gate's on / off state in the spatial dimension and achieve stable recognition of the on / off state of any type of gate, thus meeting the requirements for the recognition and detection of the gate's on / off state. Attached Figure Description

[0052] Figure 1 A flowchart illustrating the gate status determination method provided in this application embodiment;

[0053] Figure 2 A schematic diagram illustrating the process of determining the gate status provided in an embodiment of this application;

[0054] Figure 3 A schematic diagram of the structure of the first model provided in the embodiments of this application;

[0055] Figure 4 This is a schematic diagram of the structure of the second model provided in the embodiments of this application;

[0056] Figure 5Another flowchart illustrating the gate status determination method provided in this application embodiment;

[0057] Figure 6 A schematic diagram of the structure of the third model provided in the embodiments of this application;

[0058] Figure 7 This is a schematic diagram of the gate status determination device provided in the embodiments of this application;

[0059] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0060] It should be noted that the terms "first," "second," and "third" mentioned above are only used to distinguish different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0062] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0063] In the following description, the terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0064] In this application embodiment, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0065] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0066] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant national laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0067] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0068] 1) Occupied Grid Detection: This is a commonly used technique in autonomous driving and robot navigation, primarily used to build and update environmental maps so that vehicles or robots can navigate safely. The basic idea of ​​occupancy grid detection is to divide the environment into a series of grids, each grid representing an area, and to estimate whether each grid is occupied using sensor data.

[0069] 2) 3D object detection: This detection aims to predict the location, size, and category of 3D objects near autonomous vehicles and is an important functional component of the vehicle's perception system.

[0070] 3) Bird's Eye View (BEV) Features: BEV features are an image representation method that projects a scene onto a two-dimensional plane from a bird's-eye view. In the field of autonomous driving, the BEV feature fusion paradigm is mainly used to fuse feature information from multiple sensors (such as LiDAR, cameras, radar, etc.) to obtain more comprehensive and accurate environmental perception.

[0071] In related technologies, vehicles typically use a single type of sensor and a small number of sensors to detect the status of the turnstile, thereby determining whether the turnstile is open or closed. For example, a camera is used to capture images and image recognition is used to determine the open / closed status of the turnstile, or distance detection is used to determine whether a turnstile is blocking the vehicle.

[0072] However, in environments with insufficient lighting or inclement weather, the images acquired using the above methods are not clear enough. In such cases, image recognition or simple distance detection cannot accurately determine the status of the turnstile. For example, for turnstiles with lifting bars, misjudgments of the lifting bar's status are prone to occur. Therefore, the above turnstile status determination scheme cannot achieve diverse detection of the lifting status of both bar-shaped and surface-shaped turnstiles, resulting in insufficient stability and accuracy, and failing to meet the actual needs of vehicles for turnstile status detection.

[0073] To address the aforementioned technical issues, related technologies have also provided solutions that rely solely on lidar for gate detection. However, these solutions suffer from poor stability and cannot meet the status detection requirements for all types of gates.

[0074] Based on the above technical problems, this application provides a method, apparatus, equipment, medium, and program product for determining the status of a turnstile.

[0075] Figure 1 This is a flowchart illustrating the gate status determination method provided in an embodiment of this application. Figure 1 As shown, the method may include the following steps:

[0076] S101. If the distance between the vehicle and the gate is less than or equal to the distance threshold, collect image data set through the vehicle's surround view system.

[0077] The surround view system includes at least two image acquisition devices; the image data in the image dataset includes at least the features of the turnstile.

[0078] Correspondingly, if the distance between the vehicle and the gate is greater than the distance threshold, then the image data set does not need to be collected.

[0079] In one embodiment, an image acquisition device installed on the front side of the vehicle can acquire an image of the gate, including the gate itself. Based on the gate image and coordinate transformation relationships, the distance between the vehicle and the gate can be determined. The coordinate transformation relationships can include a one-to-one correspondence between the camera coordinates of the image acquisition device, the image coordinates of the gate image, and the world coordinate system.

[0080] In one embodiment, the image acquisition devices in the surround view system can be respectively installed on the front, rear, left, and right sides of the vehicle body. Exemplarily, the image acquisition devices in the surround view system can include a first set of devices and a second set of devices. Exemplarily, the image acquisition devices in the first set of devices can include four fisheye cameras, which can be respectively installed at the front, rear, left, and right sides of the vehicle body, and the field of view of the fisheye cameras can be 190 degrees. Exemplarily, the image acquisition devices in the second set of devices can include cameras installed at the front, rear left, and rear right sides of the vehicle body, each with a field of view of 120 degrees.

[0081] In one implementation, the image dataset may include a set of image data acquired by seven cameras in a surround-view system.

[0082] In one embodiment, when a vehicle approaches the gate body, the image acquisition device in the surround view system can be controlled to acquire an image data set; for example, the gate may include a gate control device and a gate body, wherein the gate control device is used to control the position and orientation of the gate body to allow or block vehicles, and the gate body may be rod-shaped, surface-shaped, or grid-shaped.

[0083] S102. Collect radar point cloud data using the vehicle's lidar.

[0084] The radar point cloud data includes at least the characteristics of the turnstile.

[0085] In one implementation, the lidar can be of the 128-thread type.

[0086] In one implementation, the radar point cloud data may include point cloud data captured by lidar corresponding to the transmitted beam.

[0087] In one implementation, radar point cloud data can be acquired through the following methods:

[0088] When a vehicle approaches the gate, the lidar sends out a laser pulse, which is reflected by the gate. The lidar can then capture the reflected laser pulse, which can be used as radar point cloud data.

[0089] S103. Based on the image data set and radar point cloud data, determine the opening and closing status of the gate.

[0090] In one implementation, the opening and closing state of the turnstile can be reflected by the posture and / or position of the turnstile body; for example, if the turnstile body is in a horizontal state and the distance between the turnstile body and the ground is less than the height of the vehicle, the opening and closing state can include the vehicle blocking state; if the distance between the turnstile body and the ground is greater than the height of the vehicle, the opening and closing state can include the vehicle allowing state.

[0091] In one implementation, the opening / closing state of the turnstile can be determined in the following way:

[0092] Feature extraction is performed on the image data in the image dataset to obtain image features. The first posture of the gate body is determined based on the image features. The radar point cloud data is analyzed to determine the second posture of the gate body. If the first posture and the second posture match, and the first posture indicates that the gate body is raised, the gate is determined to be in the vehicle release state. Conversely, if the first posture and the second posture do not match, the gate is determined to be in the vehicle blockage state.

[0093] As can be seen from the above, the gate state determination method provided in this application embodiment, since the surround view system includes at least two image acquisition devices, when the distance between the vehicle and the gate is less than or equal to a distance threshold, uses the image data set acquired by the vehicle's surround view system. Since the image data set includes features of the gate, it not only controls the operation of acquiring image data by the surround view system, but also allows for a comprehensive and high-precision characterization of the gate's position and attitude relative to the vehicle. Furthermore, the radar point cloud data acquired by the vehicle's lidar, which also includes features of the gate, allows for the determination of the gate's position and attitude from various angles. The three-dimensional space accurately represents the spatial distance and position of the gate relative to the vehicle. Based on this, the opening and closing status of the gate is determined using image data sets and radar point cloud data. This allows the gate's opening and closing status to be associated with the image data sets collected by the surround view system and the radar point cloud data collected by the lidar. It also achieves the fusion of data collected separately by the surround view system and lidar, which can improve the comprehensiveness and accuracy of the vehicle's perception of the gate and its surrounding environment. This can improve the accuracy of the gate's opening and closing status in the spatial dimension and achieve stable recognition of the opening and closing status of any type of gate, thus meeting the requirements for the recognition and detection of the gate's opening and closing status.

[0094] Based on the foregoing embodiments, the gate status determination method provided in this application, which determines the gate's open / closed status based on image data sets and radar point cloud data, can be implemented through the following steps:

[0095] SA1. Extract features from the image data in the image dataset to obtain the first image features.

[0096] In one implementation, the number of image features in the first image feature set can be matched with the number of image data in the image dataset.

[0097] In one implementation, the first image feature can be obtained in the following way:

[0098] The image data in the image dataset is preprocessed to remove redundant images and obtain a preprocessed image. Then, the preprocessed image is used to extract features from the backbone network to obtain the first image features. The redundant images may include images with a sharpness less than or equal to a sharpness threshold.

[0099] SA2. Extract features from radar point cloud data to obtain point cloud features.

[0100] In one implementation, point cloud features may include the spatial position and / or spatial distance of the gate body relative to the vehicle.

[0101] In one implementation, point cloud features can be obtained in the following way:

[0102] The point cloud features are obtained by extracting features from the radar point cloud data using a Lidar Encoder.

[0103] It should be noted that the execution order of SA1 and SA2 can be adjusted sequentially or executed in parallel, and this application embodiment does not limit this.

[0104] SA3. Perform 3D target detection based on the first image features and point cloud features to obtain the body pose of the gate body.

[0105] The turnstile includes a turnstile control device and a turnstile body; the turnstile control device is used to control the posture of the turnstile body in order to intercept or allow vehicles to pass.

[0106] In one embodiment, the gate control device can be installed in a gate housing that is fixed to the ground, and the gate housing and the gate body can be mechanically connected. The gate control device can control the opening and closing state of the gate body according to the opening and closing commands it receives.

[0107] In one embodiment, the body posture may include the first posture and the second posture in the foregoing embodiments, and may also include a third posture; wherein, the third posture may include a state in which the gate body is suspended due to an abnormal state of the gate body or the gate control device, at which time, the distance between the gate body and the ground may be greater than or equal to a first threshold and less than or equal to a second threshold; wherein, the first threshold may include the distance between the bottom surface of the gate body and the ground when the gate body is in a horizontal, unlifted state, and the second threshold may include the vehicle height.

[0108] In one implementation, the body posture may include whether the gate body is in a raised state.

[0109] In one implementation, the body posture can be determined in the following way:

[0110] The first image features are processed by the convolutional neural network contained in the gate detection model set in the vehicle to determine the position of the gate body. Then, the body posture is determined based on the vertical distance between the body position and the ground. For example, if the vertical distance is greater than a second threshold, the local posture can be determined to be an raised state. The local position can be represented by the coordinates (x, y, z) of the center point of the gate body in the world coordinate system and the yaw angle.

[0111] In one implementation, the process of determining the body posture through three-dimensional target detection can also be called 3D gate detection. It should be noted that the scope of 3D gate detection can be a first spatial range. For example, the first spatial range can include the range of 21.4m in front of the vehicle body, 10.6m behind, 11.2m to the left, and 11.2m to the right.

[0112] SA4. Based on the first image features and point cloud features, perform occupancy grid detection to obtain the relative positional relationship between the gate body and the vehicle.

[0113] In one implementation, the relative positional relationship can characterize whether the gate body occupies the space in the direction of vehicle travel from the three-dimensional spatial dimension. In other words, the relative positional relationship can characterize whether the gate body is in a state of blocking vehicles, or whether it will block at least part of the vehicle.

[0114] In one implementation, the relative positional relationship can be obtained in the following way:

[0115] The occupancy (OCC) detection model installed in the vehicle performs occupancy grid detection based on the first image features and point cloud features to predict the spatial area occupied by the gate body in three-dimensional space, and then determines the above-mentioned relative positional relationship based on the spatial area.

[0116] It should be noted that the execution order of SA3 and SA4 can be adjusted sequentially or executed in parallel; this application does not limit this.

[0117] SA5. Determine the switch state based on the body's posture and relative position relationship.

[0118] In one implementation, the switch state can be determined in the following way:

[0119] The gate's posture determines whether it is in an raised state. The relative position determines whether the gate is above the vehicle. If the gate's height is greater than the vehicle's height, the gate is in a vehicle-allowing state. If the gate's height is less than or equal to the vehicle's height, and the relative position indicates that the gate is in front of the vehicle, the gate is in a vehicle-blocking state.

[0120] As can be seen from the above, in the gate state determination method provided in this application embodiment, first image features are obtained by extracting features from image data in the image data set, and point cloud features are obtained by extracting features from radar point cloud data. In this way, comprehensive feature extraction is achieved from image data acquired by the image acquisition device and radar point cloud data captured by the lidar. Furthermore, three-dimensional target detection is performed based on the first image features and point cloud features to obtain the body posture of the gate body, thereby improving the accuracy of the body posture. At the same time, three-dimensional target detection and occupancy grid detection are performed based on the first image features and point cloud features to obtain the body posture of the gate body and the relative positional relationship between the gate body and the vehicle. Thus, through three-dimensional target detection and occupancy grid detection, comprehensive feature mining and recognition of the first image features and point cloud features can be achieved, thereby overcoming the problems of detection errors, mistakes, or insufficient accuracy that may occur with a single detection method. On this basis, the switch state is determined based on the body posture and relative positional relationship, which can improve the accuracy and stability of the switch state.

[0121] Based on the foregoing embodiments, the gate state determination method provided in this application, which determines the switch state based on the body posture and relative position relationship, can be implemented in the following ways:

[0122] If the gate's posture indicates that the gate body is in a raised state, and the relative positional relationship indicates that the gate body is not in a state of blocking vehicles, then the gate is determined to be in a state of allowing vehicles to pass.

[0123] Accordingly, if the body posture indicates that the vehicle is not in a raised state, or the relative position relationship indicates that the gate body is in a state of blocking the vehicle, then it can be determined that the gate is not in a state of allowing the vehicle to pass.

[0124] Figure 2 This is a flowchart illustrating the process of determining the gate status provided in an embodiment of this application, as shown below. Figure 2 As shown, the process may include the following steps:

[0125] S201. Acquire collected data.

[0126] For example, the collected data may include radar point cloud data and image data sets.

[0127] S202, Feature Extraction.

[0128] For example, feature extraction can be performed on radar point cloud data to obtain point cloud features, and feature extraction can also be performed on image data sets to obtain first image features.

[0129] S203, 3D Box Frame Detection.

[0130] For example, 3D Box detection can be the three-dimensional target detection described in the foregoing embodiments.

[0131] S204. Determine whether the device is in a raised state.

[0132] For example, it can be determined whether the gate body is in a raised state based on the detection results of the 3D Box detection.

[0133] For example, if it is determined that the gate body is in the raised state, S207 can be executed; if the gate body is not in the raised state, S202 can be executed.

[0134] S205, Occupied grid detection.

[0135] For example, the relative positional relationships in the foregoing embodiments can be obtained through grid occupancy detection.

[0136] S206. Determine if there are any suspended obstacles.

[0137] For example, a suspended obstacle may include a gate body that is in a state of blocking a vehicle; wherein the presence of a suspended obstacle may correspond to the gate body being in a state of blocking a vehicle.

[0138] For example, if it is determined that there is no suspended obstacle, S207 can be executed; if it is determined that there is a suspended obstacle, S202 can be executed.

[0139] It should be noted that the execution order of the first branch corresponding to S203 to S204 and the second branch corresponding to S205 to S206 can be adjusted sequentially or executed in parallel. This application embodiment does not limit this.

[0140] S207. Confirm the gate is open.

[0141] For example, opening the gate may include the gate body being in a vehicle release state.

[0142] S208, Subsequent processing procedures.

[0143] For example, the subsequent processing flow may include a data processing flow that displays an image of the gate opening or provides a voice prompt indicating that the gate is open.

[0144] For example, when a vehicle approaches the gate body, a parking operation can be automatically performed, and the opening and closing status of the gate body and changes in the surrounding environment of the gate body can be continuously tracked and monitored during the vehicle parking period. When it is determined that the gate body is in the vehicle release state and there are no suspended obstacles in front of the vehicle, it can be determined that the gate body is in the raised state. At this time, the spatial position and raised state of the gate body can be transmitted to the subsequent data processing process so that the subsequent data processing process can control the vehicle to start and pass through the gate body.

[0145] The above process allows the determination of whether the gate is open to be simultaneously linked to whether the gate is in a raised state and whether there are any suspended obstacles, including the gate itself, thereby improving the accuracy and stability of the gate's opening status.

[0146] As can be seen from the above, in the gate state determination method provided in this application embodiment, if the body posture indicates that the gate body is in the raised state and the relative position relationship indicates that the gate body is not in the state of blocking vehicles, then the gate is determined to be in the vehicle release state. This allows the gate to be in the vehicle release state to be associated with the gate body's raised state and the state of not blocking vehicles at the same time, thereby reducing the probability of misjudging the vehicle release state and improving the accuracy of the gate being in the vehicle release state.

[0147] Based on the foregoing embodiments, the gate state determination method provided in this application, which performs three-dimensional target detection based on the first image features and point cloud features to obtain the body posture of the gate body, can be achieved through the following steps:

[0148] SC1. The first image features and point cloud features are fused to obtain the first fusion result.

[0149] In one implementation, the first fusion result may include BEV features.

[0150] In one implementation, the first fusion result can be obtained in the following way:

[0151] The first image features are transferred into the BEV space to obtain BEV image features. Then, the BEV image features and point cloud features are fused using the BEV Encoder to obtain the first fusion result.

[0152] SC2. Perform feature extraction processing on the first fusion result to obtain the first processing result.

[0153] The first processing result includes at least the body posture and confidence level of the gate body.

[0154] In one implementation, the confidence level may include the probability that the entity object within the 3D detection frame is the gate itself; for example, the first processing result may be achieved through a first model.

[0155] Figure 3 A schematic diagram of the structure of the first model provided in the embodiments of this application, as shown below. Figure 3 As shown, the first model 300 may include: a first input unit, a 3×3conv, a BatchNorm, a ReLU, and a first output unit; wherein, the first input unit is used to input the first fusion result, i.e., the BEV feature, into the 3×3conv for feature extraction, to obtain the fused feature extraction result, then the BatchNorm normalizes the fused feature extraction result to obtain the normalized result, then the ReLU activates the normalized result, and finally the first output unit outputs the first processing result, including confidence, gate size and orientation, and whether the body is raised; wherein, the gate size and orientation may include data such as the position, state, and rotation angle of the gate body, and whether the body is raised can characterize whether the gate body is in a raised body posture.

[0156] For example, the above method can also perform the following operations:

[0157] The gate itself is verified based on confidence level.

[0158] In one implementation, if the confidence level is greater than or equal to the confidence threshold, the entity detected in the first processing result is determined to be the gate entity.

[0159] As can be seen from the above, in the gate state determination method provided in this application embodiment, the first image features and point cloud features are fused to obtain the first fusion result, thereby realizing the fusion of the first image features and point cloud features and improving the comprehensiveness and completeness of the features in the first fusion result; furthermore, feature extraction processing is performed on the first fusion result to obtain the first processing result including the body posture of the gate body and the confidence level, and the gate body is confirmed based on the confidence level. This not only achieves accurate detection of the body posture, but also improves the accuracy of gate body detection by adding the confidence level factor.

[0160] Based on the foregoing embodiments, the gate status determination method provided in this application embodiment, which performs occupancy grid detection based on the first image features and point cloud features to obtain the relative positional relationship between the gate body and the vehicle, can be achieved through the following steps:

[0161] SE1. The first image features and point cloud features are fused to obtain the first fusion result.

[0162] SE2. The first fusion result is processed by the grid occupancy algorithm to obtain the second processing result.

[0163] The second processing result includes at least the type and occupancy status of the voxel corresponding to the gate body.

[0164] In one implementation, the type of voxel may include movable objects, immovable objects, and gate bodies, etc.

[0165] In one implementation, the occupancy status may include whether the space corresponding to a single voxel or a set of voxels is occupied; wherein, the set of voxels may include the set of voxels within the spatial range corresponding to the first fusion result.

[0166] In one implementation, the second processing result can be obtained in the following way:

[0167] The first fusion result is converted into a voxel set by the voxel processing unit, and the three-dimensional space corresponding to the voxel set is predicted to be occupied and the type of voxel by the prediction occupancy field unit.

[0168] In one embodiment, the second processing result can be obtained through the OCC module, and the perception range of the OCC module can be a second spatial range. For example, the second spatial range can include the spatial range formed by the front side of the vehicle body 40.6m, the rear side 10.6m, the left side 11.2m, and the right side 11.2m.

[0169] For example, the second processing result can be obtained by processing the BEV features using a second model. Figure 4 This is a schematic diagram of the structure of the second model provided in the embodiments of this application, such as... Figure 4As shown, the second model 400 may include: a second input unit, a voxel processing layer, a predicted occupancy field, a voxel classification layer, and a second output unit; wherein, the second input unit is used to input BEV features to the voxel processing layer, so that the voxel processing layer converts the grid corresponding to the BEV features into a voxel set; the predicted occupancy field is used to predict whether the three-dimensional space corresponding to the voxels in the voxel set is occupied; and the voxel classification layer is used to classify the voxels in the occupied state; the second output unit is used to output the voxel classification result and the position corresponding to the voxel; wherein, the voxel classification result may include movable objects, immovable objects, and the gate body, etc.

[0170] SE3. Based on the type and occupancy status in the second processing result, determine the relative positional relationship.

[0171] In one implementation, the relative positional relationship can be determined in the following way:

[0172] Based on the type in the second processing result, determine whether the space corresponding to the voxel or voxel set contains the gate body. Based on the spatial position of the voxel or voxel set corresponding to the occupancy status, determine the position of the gate body. Then, based on the position of the body and the position of the vehicle, determine the relative positional relationship.

[0173] As can be seen from the above, in the gate state determination method provided in this application embodiment, the first image features and point cloud features are fused to obtain the first fusion result, which realizes the comprehensive fusion of image features and point cloud features, thereby improving the integrity and comprehensiveness of the features in the first fusion result; and, the first fusion result is processed by the occupancy grid algorithm to obtain the second processing result, which at least includes the type and occupancy status of the voxel corresponding to the gate body. Thus, the second processing result can be used to characterize the spatial position of the gate body in three-dimensional space with finer granularity; on this basis, the relative positional relationship is determined based on the type and occupancy status in the second processing result, thereby improving the accuracy of the relative positional relationship.

[0174] Based on the foregoing embodiments, the gate state determination method provided in this application, which fuses the first image features and point cloud features to obtain the first fusion result, can be implemented in the following ways:

[0175] The first image features and point cloud features are fused in the BEV space to obtain the third processing result; the third processing result is then processed by an encoder to obtain the first fusion result.

[0176] In one implementation, the third processing result can be obtained in the following way:

[0177] The first image features are transformed into the BEV space, and then the transformed first image features and point cloud features are fused in the BEV space to obtain the third processing result.

[0178] In one implementation, the third processing result can be processed by the encoder of the BEV to obtain the first fusion result.

[0179] As can be seen from the above, the gate state determination method provided in this application performs fusion processing on the first image features and point cloud features in the BEV space to obtain the third processing result, and then processes the third processing result through an encoder to obtain the first fusion result. In this way, the comprehensiveness and completeness of the features carried in the first fusion result can be improved.

[0180] Figure 5 Another flowchart illustrating the gate state determination method provided in this application embodiment is shown below. Figure 5 As shown in the flowchart 500, the method for determining the status of the turnstile may include:

[0181] By acquiring data using a 7V camera, an image dataset can be obtained; wherein, the 7V camera may include the surround view system in the aforementioned embodiments.

[0182] By extracting features from the image data in the image dataset using Backbone, the first image features can be obtained; then, by processing the image features based on camera parameters using View Transform, the image features can be transformed into the BEV space.

[0183] By capturing data with LiDAR, radar point cloud data can be obtained; by performing feature extraction processing on the radar point cloud data using LiDAR Encoder, point cloud features can be obtained; and then the point cloud features are converted into BEV space to obtain LiDAR BEV Features.

[0184] For example, the first image features converted to BEV space and LidarBEV Features can be processed by BEV Encoder to obtain the first fusion result, namely Fused BEV Features. Then, gate detection and occupancy grid detection are performed based on Fused BEV Features to obtain the body pose and relative position relationship. Finally, the opening and closing state of the gate body can be determined based on the body pose and relative position relationship.

[0185] For example, gate detection may include spatial target detection as described in the foregoing embodiments.

[0186] In the above process, the occupancy grid detection can accurately perceive the space occupied by the gate body and the occupancy of its surrounding space, thereby improving the comprehensiveness and accuracy of the detection of the gate body and its surrounding environment.

[0187] Furthermore, in practical applications, different turnstile bodies may have different shapes and sizes. Occupation grid detection, based on the principle of space occupancy, can capture the shape and position changes of rod-shaped and planar turnstile bodies. It can obtain information about the turnstile from different angles. Therefore, through occupancy grid detection, not only can the position of the turnstile body be determined, but also the spatial distance between the turnstile body and the vehicle, as well as whether there are other obstacles around the turnstile body. This provides a more comprehensive and accurate basis for vehicle passage decisions, thereby improving the stability of turnstile body recognition and spatial shape determination.

[0188] On the other hand, in the above process, the combination of occupancy grid detection and gate detection forms a complementary gate status detection system. The spatial target detection represented by gate detection can determine the body posture of the gate body in the 3D box frame, while occupancy grid detection can realize the perception of the specific suspended position of the gate body. Therefore, by combining the two, the deviation or error of gate detection can be compensated, thereby improving the accuracy and reliability of gate status detection.

[0189] Based on the foregoing embodiments, the gate status determination method provided in this application can also perform the following steps:

[0190] SG1. If radar point cloud data is not available, perform feature extraction processing on the image data set to obtain the second image features.

[0191] Accordingly, if radar point cloud data is obtained, the opening and closing status of the gate can be determined by the method provided in the aforementioned embodiments.

[0192] In one implementation, it can be determined that radar point cloud data cannot be obtained if any of the following conditions are met:

[0193] The lidar is disabled.

[0194] The lidar is enabled, but the vehicle is currently configured to not use radar point cloud data to determine the gate's opening and closing status.

[0195] The lidar is malfunctioning;

[0196] The lidar is functioning normally, but the communication connection between the lidar and the computing module in the vehicle used to perform gate status detection is abnormal.

[0197] In one implementation, a second image feature can be obtained by performing feature extraction processing on image features in an image dataset using a feature extraction model that includes a convolutional neural network and an image encoder.

[0198] SG2, Get text data.

[0199] The text data includes at least the data on the status of the gate itself.

[0200] In one implementation, the text data may be pre-set data used to indicate at least one parameter among the shape, position, and size of the gate body for identification and extraction.

[0201] In one implementation, the text data may include the prompt word.

[0202] SG3. The text features corresponding to the text data are fused with the second image features to obtain the second fusion result.

[0203] In one implementation, the second fusion result may include the result of filtering or selecting second image features based on text features.

[0204] In one implementation, text data can be processed using a recurrent neural network language model to obtain text features.

[0205] In one implementation, the second fusion result can be obtained in the following way:

[0206] By aligning and fusing text features and second image features through attention mechanisms or Cross-Attention, visual language features are obtained and identified as the second fusion result.

[0207] SG4. The second fusion result is processed using a Vision Language Model (VLM) to determine the opening and closing status of the gate.

[0208] It's worth noting that VLM can simultaneously process and understand multimodal information, including image and text features. This makes VLM models perform exceptionally well in scenarios such as image caption generation and visual question answering. Furthermore, because VLM's training process is based on a large amount of diverse sample data, it can make reasonable decisions in tasks where training is not explicitly defined. Simultaneously, through parameter or model fine-tuning, VLM can achieve seamless coupling with different downstream tasks, enabling its application to specific use cases or domains. The drawbacks of VLM are that its training requires significant computational resources and abundant multimodal sample data, and the complexity of its model architecture makes its understanding and interpretation difficult, thus posing challenges to VLM's development, debugging, and optimization.

[0209] In one implementation, the switch state can be determined in the following way:

[0210] The fusion vector contained in the second fusion result is processed by VLM to generate visually perceived text, and the state represented by the visually perceived text is determined as a switch state.

[0211] For example, the switch state can be implemented using a third model. Figure 6 A schematic diagram of the structure of the third model provided in the embodiments of this application, as shown below. Figure 6 As shown, the third model 600 may include: a 7V camera, an image encoder, a text input unit, a text encoder, image-text fusion layers, a VLM decoder, and a third output unit.

[0212] A 7V camera is used to acquire and send image data sets to the Image Encoder, so that the Image Encoder can extract features from the image data sets to obtain second image features.

[0213] The text input unit is used to input text data into the TEXT Encoder so that the TEXT Encoder can extract features from the text data to obtain text features.

[0214] Image-Text Fusion Layers are used to fuse text features and second image features to obtain a second fusion result.

[0215] The VLM Decoder is used to process the second fusion result based on text data to obtain the switch state, which is then output by the third output unit.

[0216] As can be seen from the above, the gate state determination method provided in this application, if radar point cloud data is not obtained, performs feature extraction processing on the image data set to obtain second image features, fuses the text features corresponding to the text data with the second image features, and obtains a second fusion result. The text data includes data for detecting the state of the gate body. Thus, through the above operations, the redundancy of features in the second fusion result can be reduced, and the effectiveness of the gate body features contained in the second fusion result can be improved. On this basis, the second fusion result is processed by VLM to determine the switch state. With the advantages of VLM in multimodal processing and accuracy, accurate and stable detection of the state of gate bodies of various types and in various environments can be achieved, thereby improving the accuracy of the switch state. Furthermore, through the above operations, the stability of determining the switch state of the vehicle can be improved.

[0217] This application also provides a gate status determination device. Figure 7 This is a schematic diagram of the gate status determination device provided in the embodiments of this application, as shown below. Figure 7 As shown, the gate status determination device 700 includes:

[0218] The acquisition module 701 is used to acquire an image data set through the vehicle's surround view system and acquire radar point cloud data through the vehicle's lidar if the distance between the vehicle and the gate is less than or equal to a distance threshold. The surround view system includes at least two image acquisition devices. The image data in the image data set includes at least the features of the gate. The radar point cloud data includes at least the features of the gate.

[0219] The determination module 702 is used to determine the opening and closing status of the gate based on the image data set and radar point cloud data.

[0220] In some embodiments, the determining module 702 is further configured to perform feature extraction on the image data in the image data set to obtain a first image feature; perform feature extraction on the radar point cloud data to obtain a point cloud feature; and perform three-dimensional target detection based on the first image feature and the point cloud feature to obtain the body posture of the gate body; wherein the gate includes a gate control device and a gate body; the gate control device is used to control the body posture of the gate body to intercept or allow vehicles to pass.

[0221] The determination module 702 is also used to perform occupancy grid detection based on the first image features and point cloud features to obtain the relative positional relationship between the gate body and the vehicle; and to determine the switch state based on the body posture and the relative positional relationship.

[0222] In some embodiments, the determining module 702 is further configured to determine that the gate is in a vehicle-allowing state if the body posture characterizes that the gate body is in a raised state and the relative position relationship characterizes that the gate body is not in a state of blocking vehicles.

[0223] In some embodiments, the determining module 702 is further configured to fuse the first image features and point cloud features to obtain a first fusion result; perform feature extraction processing on the first fusion result to obtain a first processing result; wherein the first processing result includes at least the body pose and confidence level of the gate body;

[0224] The determination module 702 is also used to confirm the gate body based on the confidence level.

[0225] In some embodiments, the determining module 702 is further configured to fuse the first image features and point cloud features to obtain a first fusion result; and process the first fusion result using an occupancy grid algorithm to obtain a second processing result; wherein the second processing result includes at least the type and occupancy status of the voxel corresponding to the gate body;

[0226] The determination module 702 is also used to determine the relative positional relationship based on the type and occupancy status in the second processing result.

[0227] In some embodiments, the determining module 702 is further configured to perform fusion processing on the first image features and point cloud features in the BEV space to obtain a third processing result; and to process the third processing result by an encoder to obtain a first fusion result.

[0228] In some embodiments, the determining module 702 is further configured to perform feature extraction processing on the image data set to obtain second image features if radar point cloud data is not obtained; and acquire text data; wherein the text data includes at least data on the detection of the gate's status;

[0229] The determination module 702 is also used to fuse the text features corresponding to the text data with the second image features to obtain a second fusion result; and to process the second fusion result through a visual language model to determine the opening and closing status of the gate.

[0230] This application also provides an electronic device. Figure 8 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application, such as... Figure 8 As shown, the electronic device 800 includes a processor 801 and a memory 802; the memory 802 stores a computer program; when the computer program is executed by the processor 801, it can implement the gate state determination method as described above.

[0231] This application also provides a computer-readable storage medium storing a computer program; when the computer program is executed by the processor of an electronic device, it can implement the gate state determination method as described above.

[0232] This application also provides a computer program product, which includes a computer program; when the computer program is executed by the processor of an electronic device, it can implement the gate state determination method as described above.

[0233] In some embodiments, the computer-readable storage medium may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM), etc.; or it may be a device that includes one or any combination of the above-mentioned memories.

[0234] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0235] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

Claims

1. A method for determining the status of a turnstile, characterized in that, The method includes: If the distance between the vehicle and the gate is less than or equal to a distance threshold, an image data set is acquired through the vehicle's surround view system; wherein, the surround view system includes at least two image acquisition devices; the image data set includes at least the features of the gate; Radar point cloud data is collected by the vehicle's lidar; wherein the radar point cloud data includes at least the features of the gate. Based on the image data set and the radar point cloud data, the opening and closing status of the gate is determined.

2. The method according to claim 1, characterized in that, Determining the opening / closing status of the gate based on the image data set and the radar point cloud data includes: Feature extraction is performed on the image data in the image dataset to obtain the first image feature; Feature extraction is performed on the radar point cloud data to obtain point cloud features; Three-dimensional target detection is performed based on the first image features and the point cloud features to obtain the body posture of the gate body; wherein, the gate includes a gate control device and the gate body; the gate control device is used to control the body posture of the gate body to intercept or allow the vehicle to pass. Based on the first image features and the point cloud features, occupancy grid detection is performed to obtain the relative positional relationship between the gate body and the vehicle; The switch state is determined based on the body's posture and the relative positional relationship.

3. The method according to claim 2, characterized in that, Determining the switch state based on the body posture and the relative position relationship includes: If the body posture indicates that the gate body is in a raised state, and the relative position relationship indicates that the gate body is not in a state of blocking the vehicle, then the gate is determined to be in a vehicle-allowing state.

4. The method according to claim 2, characterized in that, The step of performing 3D target detection based on the first image features and the point cloud features to obtain the body pose of the gate body includes: The first image features and the point cloud features are fused to obtain the first fusion result; The first fusion result is subjected to feature extraction processing to obtain a first processing result; wherein, the first processing result includes at least the ontology pose and confidence level; The method further includes: The gate body is confirmed based on the confidence level.

5. The method according to claim 2, characterized in that, The step of performing occupancy grid detection based on the first image features and the point cloud features to obtain the relative positional relationship between the gate body and the vehicle includes: The first image features and the point cloud features are fused to obtain the first fusion result; The first fusion result is processed by an occupancy grid algorithm to obtain a second processing result; wherein, the second processing result includes at least the type and occupancy status of the voxel corresponding to the gate body; The relative positional relationship is determined based on the type and occupancy status in the second processing result.

6. The method according to claim 4 or 5, characterized in that, The process of fusing the first image features and the point cloud features to obtain a first fusion result includes: The first image features and the point cloud features are fused together in the bird's-eye view BEV space to obtain the third processing result; The first fusion result is obtained by processing the third processing result through an encoder.

7. The method according to claim 1, characterized in that, The method further includes: If the radar point cloud data is not obtained, feature extraction processing is performed on the image data set to obtain the second image features; Acquire text data; wherein the text data includes at least data for detecting the status of the gate; By fusing the text features corresponding to the text data with the second image features, a second fusion result is obtained; The second fusion result is processed using a visual language model to determine the opening and closing status of the gate.

8. A gate status determination device, characterized in that, The gate status determination device includes: The acquisition module is used to acquire an image data set through the vehicle's surround view system and to acquire radar point cloud data through the vehicle's lidar if the distance between the vehicle and the gate is less than or equal to a distance threshold. The surround view system includes at least two image acquisition devices. The image data set includes at least the features of the gate. The point cloud data includes at least the features of the gate. The determination module is used to determine the opening and closing status of the gate based on the image data set and the radar point cloud data.

9. An electronic device, characterized in that, The electronic device includes a processor and a memory; the memory stores a computer program; when the computer program is executed by the processor, it can implement the gate state determination method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program; when the computer program is executed by the processor of the electronic device, it can implement the gate state determination method as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, The program product includes a computer program; when the computer program is executed by the processor of an electronic device, it is able to implement the gate state determination method as described in any one of claims 1 to 7.

Images