Vehicle loopback detection method, device and equipment and storage medium

Abstract

The invention discloses a vehicle loopback detection method, device and equipment and a storage medium, and relates to the technical field of vehicle image processing, and the method comprises the steps: obtaining a current frame image collected by a target vehicle; when the target vehicle meets a preset space-time condition, pre-screening the historical frame image based on the current frame image to obtain a reference frame image; performing semantic element matching on the current frame image and the reference frame image to obtain a semantic matching result; based on the semantic matching result, loopback detection of the target vehicle is completed; according to the method, images are collected in real time and stored as historical images, when it is considered that a vehicle may have a loopback according to the driving time and the driving distance of the vehicle, semantic elements in the current image are matched with semantic elements in the historical images meeting requirements, and when matching succeeds, the current position is the same as the position when the historical images are collected; and when the vehicle loopback positioning is successful, real-time image loopback detection can be carried out according to historical images, and whether the current state is loopback or not can be accurately judged.

Description

Technical Field

[0001] The present application relates to the technical field of vehicle image processing, and in particular to a vehicle loop detection method, device, equipment and storage medium. Background Art

[0002] Visual SLAM (Simultaneous Localization and Mapping) technology is an important and indispensable technology in the field of intelligent driving. SLAM technology mainly includes two functions: positioning and mapping. Its main modules are front-end odometer, loop detection, back-end optimization and map construction. The odometer part calculates the initial posture of the carrier through the observation constraints between the previous and next moments, and then optimizes the posture by constructing the reprojection error constraints through the back-end optimization part. However, as the distance traveled increases, the cumulative error will become larger and larger, which requires loop detection to reposition the carrier, add loop constraints and then eliminate the cumulative error to obtain a globally consistent map. Therefore, loop detection is very important in the entire SLAM system.

[0003] The current mainstream solution is the loop detection algorithm based on the word band model, which mainly extracts and encodes features of a large number of scene graphs, trains the word band model, and then searches for a match in the word band when the carrier reaches a similar scene again. If the match is successful, it means that the loop detection is successful. The loop detection of the word band model will be greatly disturbed in dynamic scenes and scenes with large changes in light. Because in dynamic scenes, the scene encoding of the word band is likely to extract the features of dynamic objects and encode them into the model, and when the carrier reaches this scene again, the dynamic objects disappear or are in a different position, which makes the matching of the previous and next information inconsistent and fails. Similarly, large changes in light will also cause inconsistencies between the previous and next scenes.

[0004] In view of the limitations of the word band model, a loop detection algorithm based on deep learning has been proposed. It mainly extracts scene image features through convolutional neural networks, and then performs loop detection based on the similarity of scene features. However, its limitations are that it lacks intuitive meaning and is a black box model. Secondly, it relies on a large number of labeled data sets and the similarity between the current environment and the data set. There is still a lack of clearer guidance in the extraction of information features, and the adjustment of training parameters is also more dependent on the adjustment experience accumulated in the past. Another important point is that it is difficult to ensure the real-time operation of the algorithm in large scenes with limited computing power of the device.

[0005] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0006] The main purpose of the present application is to provide a vehicle loopback detection method, device, equipment and storage medium, aiming to solve the technical problem that the prior art cannot perform vehicle loopback detection in real time and accurately.

[0007] To achieve the above objectives, the present application proposes a vehicle loopback detection method, which includes:

[0008] Get the current frame image collected by the target vehicle;

[0009] When the target vehicle meets the preset time and space conditions, pre-screening the historical frame images based on the current frame image to obtain a reference frame image;

[0010] Performing semantic element matching between the current frame image and the reference frame image to obtain a semantic matching result;

[0011] Complete loop detection on the target vehicle based on the semantic matching result.

[0012] In one embodiment, when the target vehicle meets the preset time and space conditions, the historical frame images are pre-screened based on the current frame image to obtain the reference frame image, including:

[0013] Acquire driving parameters of the target vehicle, and obtain driving distance and driving time according to the driving parameters;

[0014] When the driving distance is greater than or equal to the distance threshold and the driving time is greater than or equal to the time threshold, the target vehicle meets the preset time and space conditions;

[0015] When the driving distance is less than the distance threshold or the driving duration is less than the duration threshold, the target vehicle does not meet the preset time and space conditions.

[0016] In one embodiment, when the driving distance is greater than or equal to a distance threshold and the driving time is greater than or equal to a time threshold, before the target vehicle meets a preset time and space condition, the method further includes:

[0017] Obtaining a reference area of ​​the garage where the target vehicle is located, and obtaining a distance threshold according to the reference area;

[0018] A duration threshold is obtained according to the driving speed parameter of the target vehicle and the reference area.

[0019] In one embodiment, when the target vehicle meets the preset time and space conditions, pre-screening the historical frame images based on the current frame image to obtain the reference frame image includes:

[0020] When the target vehicle meets the preset time and space conditions, the instance category and instance quantity of each historical frame image are matched with the instance category and instance quantity of the current frame image;

[0021] The successfully matched historical frame image is used as the reference frame image.

[0022] In one embodiment, when the target vehicle meets the preset time and space conditions, the historical frame images are pre-screened based on the current frame image to obtain the reference frame image, and the method further includes:

[0023] Get historical frame images;

[0024] Performing image perception analysis on the historical frame images to obtain semantic point cloud information;

[0025] Performing point cloud filtering on the semantic point cloud information to obtain semantic information;

[0026] Instance segmentation is performed based on the semantic information to obtain instance categories and instance quantities of the historical frame image.

[0027] In one embodiment, performing image perception analysis on the historical frame image to obtain semantic point cloud information includes:

[0028] Obtaining camera intrinsic parameters and camera extrinsic parameters of the target vehicle;

[0029] Obtaining depth information of the historical frame image according to the camera intrinsic parameters and the camera extrinsic parameters;

[0030] Identifying pixel coordinates of semantic information in the historical frame image;

[0031] Semantic point cloud information is obtained according to the depth information and the pixel coordinates.

[0032] In one embodiment, performing semantic element matching on the current frame image and the reference frame image to obtain a semantic matching result includes:

[0033] Matching point clouds with the same semantic information in the current frame image and the reference frame image to obtain a plurality of point cloud rotation matrices and point cloud translation matrices;

[0034] Performing consistency evaluation on the multiple point cloud rotation matrices and the point cloud translation matrix to obtain a comprehensive score;

[0035] A semantic matching result is obtained based on the comprehensive score.

[0036] In addition, to achieve the above purpose, the present application also proposes a vehicle loop detection device, the vehicle loop detection device comprising:

[0037] An image acquisition module is used to acquire the current frame image collected by the target vehicle;

[0038] An image screening module, used for pre-screening historical frame images based on the current frame image to obtain a reference frame image when the target vehicle meets the preset time and space conditions;

[0039] A point cloud matching module, used for performing semantic element matching between the current frame image and the reference frame image to obtain a semantic matching result;

[0040] The loop detection module is used to complete the loop detection of the target vehicle based on the semantic matching result.

[0041] In addition, to achieve the above-mentioned purpose, the present application also proposes a vehicle loop detection device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the vehicle loop detection method described above.

[0042] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the vehicle loop detection method described above are implemented.

[0043] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the vehicle loop detection method described above are implemented.

[0044] One or more technical solutions proposed in this application have at least the following technical effects:

[0045] The vehicle image acquisition device collects images in real time and stores them as historical images. When it is considered that the vehicle may be in a loop based on the vehicle's driving time and driving distance, the semantic elements in the current image are matched with the semantic elements in the historical images that meet the requirements. When the match is successful, the current position is the same as the position when the historical image was collected, and the vehicle loop positioning is successful. The real-time image loop detection can be performed based on the historical image to accurately determine whether it is currently in a loop. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0047] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0048] Figure 1 A schematic diagram of a process flow provided for the first embodiment of the vehicle loop detection method of the present application;

[0049] Figure 2 A schematic diagram of a limited range of loop detection provided in Example 1 of the vehicle loop detection method of the present application;

[0050] Figure 3 A schematic diagram of a flow chart provided for the second embodiment of the vehicle loop detection method of the present application;

[0051] Figure 4 This is a schematic diagram of the module structure of the vehicle loop detection device according to an embodiment of the present application;

[0052] Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the vehicle loopback detection method in the embodiment of the present application. DETAILED DESCRIPTION

[0053] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0054] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0055] The main solution of the embodiment of the present application is: obtain the current frame image collected by the target vehicle; when the target vehicle meets the preset time and space conditions, pre-screen the historical frame images based on the current frame image to obtain a reference frame image; match the semantic elements of the current frame image and the reference frame image to obtain a semantic matching result; and complete loop detection of the target vehicle based on the semantic matching result.

[0056] In this embodiment, for ease of description, the following description is made by taking the identification of a vehicle loop detection device as the execution subject.

[0057] The current mainstream solutions in the prior art include loop detection algorithms based on word band models, which mainly extract and encode features from a large number of scene graphs to train word band models. Then, when the carrier reaches a similar scene again, it searches for a pair in the word band. If the pairing is successful, it means that the loop detection is successful. The loop detection of the word band model will be greatly disturbed in dynamic scenes and scenes with large changes in light. Because in dynamic scenes, the scene encoding of the word band is likely to extract the features of dynamic objects and incorporate them into the model. When the carrier reaches the scene again, the dynamic objects disappear or are in a different position, which causes the inconsistent matching of the previous and next information to fail. Similarly, large changes in light will also cause inconsistencies between the previous and next scenes.

[0058] The present application provides a solution, which collects images in real time and stores them as historical images. When it is believed that the vehicle may be in a loop based on the vehicle's driving time and driving distance, the semantic elements in the current image are matched with the semantic elements in the historical images that meet the requirements. When the match is successful, the current position is the same as the position when the historical image was collected, and the vehicle loop positioning is successful. The real-time image loop detection can be performed based on the historical image to accurately determine whether it is currently in a loop.

[0059] It can be seen from the above embodiments that the present application discloses a vehicle loop detection method, device, equipment and storage medium, which relates to the field of vehicle image processing technology, and discloses: obtaining a current frame image collected by a target vehicle; when the target vehicle meets preset time and space conditions, pre-screening the historical frame image based on the current frame image to obtain a reference frame image; matching the semantic elements of the current frame image and the reference frame image to obtain a semantic matching result; completing loop detection of the target vehicle based on the semantic matching result; the method collects images in real time and stores them as historical images. When it is considered that the vehicle may have a loop based on the vehicle's driving time and driving distance, the semantic elements in the current image are matched with the semantic elements in the historical images that meet the requirements. When the match is successful, the current position is the same as the position when the historical image was collected, and the vehicle loop positioning is successful. The real-time image loop detection can be performed based on the historical image to accurately determine whether it is currently in a loop.

[0060] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a vehicle loop detection device, etc. The following takes the vehicle loop detection device as an example to illustrate this embodiment and the following embodiments.

[0061] Based on this, the present application embodiment provides a vehicle loop detection method, referring to Figure 1 , Figure 1This is a flow chart of the first embodiment of the vehicle loop detection method of the present application.

[0062] In this embodiment, the vehicle loop detection method includes steps S10 to S40:

[0063] Step S10, obtaining a current frame image captured by the target vehicle.

[0064] It is understandable that the target vehicle may be a vehicle that needs to undergo loop detection, and the current frame image may be acquired through an image acquisition device that is a peripheral of the vehicle.

[0065] It should be noted that the vehicle is set with a storage space for storing images. The vehicle collects external images in real time. Each image collected is a frame image. The current frame image collected in real time is stored in the storage space. The image in the storage space can be used as a historical frame image.

[0066] It should be emphasized that the historical frame images in the storage space may be stored in advance in the order of storage time, so as to facilitate subsequent screening of the historical frame images.

[0067] Step S20, when the target vehicle meets the preset time and space conditions, pre-screen the historical frame images based on the current frame image to obtain a reference frame image.

[0068] It should be noted that the preset time and space conditions may be time threshold conditions and distance threshold conditions set according to the size of the space for loop detection, the vehicle's driving speed, and the vehicle's driving distance.

[0069] It should be noted that filtering the historical frame images according to the current frame image may be filtering the historical frame images according to the type and amount of semantic information in the current frame image.

[0070] It should be noted that the reference frame image may be an image in the historical images that has the same type of semantic information and the same amount of semantic information as the current frame image.

[0071] In a feasible implementation manner, step S20 may include steps A201 to A203:

[0072] Step A201, obtaining the driving parameters of the target vehicle, and obtaining the driving distance and driving time according to the driving parameters.

[0073] It is understandable that the driving parameters of the target vehicle may be the vehicle speed, the vehicle entry time point, the vehicle entry position, etc.

[0074] It should be understood that the vehicle itself will detect the vehicle's real-time driving speed, and there may be a timestamp inside the vehicle to record the time. If the network is connected, the vehicle's entry time can be directly obtained. The vehicle can record the real-time position, and can directly use a certain position in the historical position as the entry position, or determine the entry position based on the user's input information.

[0075] It should be noted that obtaining the driving distance and driving duration according to the driving parameters may be obtaining the driving duration according to the driving entry time and the current time, or obtaining the driving distance according to the driving duration and the driving speed.

[0076] Step A202, when the driving distance is greater than or equal to the distance threshold and the driving time is greater than or equal to the time threshold, the target vehicle meets the preset time and space conditions.

[0077] It should be noted that the judgment on whether the target vehicle meets the preset time and space conditions is based on the fact that during the operation of the Slam system, the odometer will calculate the position (position and attitude) of the carrier in real time. When the carrier returns to the original position, it should first meet two conditions: being near the position of the loop frame; and being relatively far away in time.

[0078] It should be emphasized that the pre-set time and space conditions limit the scope of loop retrieval from two dimensions to avoid unnecessary calculations; the dimensions of limiting the search range are divided into two: 1. In the spatial dimension, because the longer the carrier runs, the longer the cumulative error will be, and an adaptive position distance limit is set according to the accuracy of the odometer and the length of the mileage; 2. In the temporal dimension, according to the distance of time and the speed of the carrier, a time distance threshold is set to exclude the range of relatively close running time. For details, please refer to Figure 2 , Figure 2 In the diagram for limiting the range of loop detection, when the carrier moves from position X12 to position X13, it may actually return to the vicinity of the starting position. However, due to the accumulated error, it does not completely overlap with the initial position, but is still near the starting position. Therefore, when performing loop detection, you can search in key frames that are near the trajectory but relatively far in time.

[0079] It should be noted that when the driving distance is greater than or equal to the distance threshold and the driving duration is greater than or equal to the duration threshold, before the target vehicle meets the preset time and space conditions, it also includes: obtaining a reference area of ​​the garage where the target vehicle is located, and obtaining a distance threshold based on the reference area; and obtaining a duration threshold based on the driving speed parameters of the target vehicle and the reference area.

[0080] Among them, it is understandable that the distance threshold and the duration threshold can be determined according to the area size of the target vehicle entering the area. For example, when a vehicle loop detection is performed when a vehicle enters an underground garage, the distance threshold and the duration threshold can be determined according to the area of ​​the underground garage or according to the length and width of the underground garage.

[0081] It should be noted that the duration threshold can be set based on the user's driving distance and the circumference of the vehicle. For example, when the vehicle's driving distance is greater than or equal to the distance threshold (90% of the basement circumference, 100% of the circumference, etc.), the current frame image and the historical frame image are matched.

[0082] It should be noted that the duration threshold may be determined based on the vehicle's driving speed. The duration threshold may be determined based on the vehicle's driving speed. The duration threshold may be determined based on the vehicle's driving speed.

[0083] Step A203, when the driving distance is less than the distance threshold or the driving duration is less than the duration threshold, the target vehicle does not meet the preset time and space conditions.

[0084] It is understandable that a driving distance less than the distance threshold or a driving duration less than the duration threshold can be understood as having just entered an area such as a basement where loop detection is required. At this time, the driving time is short and the driving distance is too close, and the possibility of a loop is low. Therefore, historical frame images are not screened, and thus loop detection is not performed, thereby avoiding redundant data calculation.

[0085] In this implementation, the scope of the loop search is limited by using the trajectory information and time information of the odometer, and alignment is performed within the search range based on semantic information, thereby improving the robustness and accuracy of the traditional loop detection algorithm while ensuring the real-time and controllability of the algorithm.

[0086] The above are merely feasible implementations before step S20 provided in this embodiment, and this embodiment does not specifically limit the specific implementations before step S20.

[0087] Step S30, performing semantic element matching between the current frame image and the reference frame image to obtain a semantic matching result.

[0088] It is understandable that the semantic matching result may include a matching failure and a matching success.

[0089] It should be noted that the semantic element matching of the current frame image and the reference frame image can be performed by matching the arrow (semantic element, or semantic elements such as signboards and indicator lights) in the current frame image with the same semantic element arrow in the reference frame image.

[0090] It should be noted that the matching of the current frame image and the reference frame image may be achieved through a three-dimensional model or point cloud data registration technology, such as an ICP (Iterative Closest Point) algorithm.

[0091] In a feasible implementation, step S30 may include steps A31 to A33:

[0092] Step A31, matching the point clouds with the same semantic information in the current frame image and the reference frame image to obtain a plurality of point cloud rotation matrices and point cloud translation matrices.

[0093] It should be noted that there are ground features such as arrows and parking spaces in the current frame image. When performing ICP matching, the arrow point cloud in the current frame will only be matched with the arrow point cloud in the candidate frame, and the parking point cloud in the current frame will be matched with the parking point cloud in the candidate frame. Traditional ICP will match all point clouds in the current frame with all point clouds in the candidate frame without distinguishing semantic information, which will increase the probability of incorrect matching of different features.

[0094] It is understandable that if there is only one point cloud with the same semantic information in the current frame image and the reference frame image, there is only one point cloud rotation matrix and point cloud translation matrix; if there are multiple point clouds with the same semantic information, there are multiple point cloud rotation matrices and point cloud translation matrices.

[0095] It should be noted that the point clouds with the same semantic information in the current frame image and the reference frame image are matched to obtain multiple point cloud rotation matrices and point cloud translation matrices, which can refer to the following formulas, so that the goal is to minimize the following error function:

[0096]

[0097] Among them, pi represents the point cloud of the semantic elements in the previous frame, qi represents the point cloud of the semantic elements in the current frame, R is the rotation matrix, which describes the rotation between point clouds, t is the translation vector, which describes the translation between point clouds, and N is the total number of different feature categories in each frame image.

[0098] Furthermore, an ICP match is performed on the point clouds with the same semantic elements in each group (such as the arrow in the previous frame and the arrow in the current frame), as shown in the following formula:

[0099] E(Ri,ti)=||PiT-Ri q i-ti||2

[0100] It can be understood that by using the iterative optimization method to find the minimum value of the residual equation, the corresponding transformation matrices Ri and ti can be obtained, and then the R and t obtained above can be subjected to consistency check.

[0101] Step A32, performing consistency evaluation on the multiple point cloud rotation matrices and the point cloud translation matrix to obtain a comprehensive score.

[0102] It is understandable that if a loop occurs, the rotation matrix and translation matrix of each semantic element in the current frame image and each semantic element in the historical frame are theoretically the same, and the consistency of each rotation matrix is ​​evaluated based on this standard.

[0103] It should be understood that because point clouds process different semantic elements, there may be deviations. In specific implementations, since point clouds with different semantic information obtained in the same frame of the picture should have undergone the same rotation and translation transformation and mapped to the next frame of the picture, theoretically the rotation and translation transformations obtained by the two methods should be the same. Using this principle, each Ri, ti obtained is given a corresponding weight score. A high score is given to a high similarity with R, t, and a low score is given otherwise. Finally, the scores are added together, and the candidate frame with the highest comprehensive score is the loop frame, so that the loop detection is successful. The formula for obtaining the comprehensive score is as follows:

[0104]

[0105] Among them, Ti represents rotation and translation Ri, ti, and S represent weights.

[0106] Step A33, obtaining a semantic matching result based on the comprehensive score.

[0107] It is understandable that a scoring threshold is preset, and when the comprehensive score is greater than the scoring threshold, the semantic matching result is a successful match; if it is less than or equal to the scoring threshold, the semantic matching result is a failed match.

[0108] In this embodiment, images with the same semantic information category and the same semantic information are obtained by pre-screening, and then ICP matching is performed on the same elements in the same images respectively. This can ensure that only point clouds with the same semantic information will be matched together, reducing the probability of false matching.

[0109] The above are only feasible implementations of step S30 provided in this embodiment, and this embodiment does not specifically limit the specific implementation of step S30.

[0110] Step S40, completing loop detection on the target vehicle based on the semantic matching result.

[0111] It should be noted that if the semantic matching result is a successful match, the loop detection is considered to be successful; if the semantic matching result is a failed match, it can be understood that no loop point has been matched yet.

[0112] This embodiment provides a vehicle loop detection method, which uses a vehicle image acquisition device to collect images in real time and store them as historical images. When it is considered that the vehicle may be in a loop based on the vehicle's driving time and driving distance, the semantic elements in the current image are matched with the semantic elements in the historical images that meet the requirements. When the match is successful, the current position is the same as the position when the historical image was collected, and the vehicle loop positioning is successful. The real-time image loop detection can be performed based on the historical images to accurately determine whether it is currently in a loop.

[0113] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 3 , step S20, the vehicle loop detection method further includes steps S21-S22:

[0114] Step S21, when the target vehicle meets the preset time and space conditions, the instance category and instance quantity of each historical frame image are matched with the instance category and instance quantity of the current frame image.

[0115] It is understandable that when the vehicle collects images in real time, when the current frame image is stored, semantic information recognition can be performed on the current frame image to obtain the semantic information in each frame image, thereby obtaining the instance category and the number of instances.

[0116] It should be understood that instances can also be understood as semantic elements, namely arrows, road signs, signboards, etc. in each frame of the image.

[0117] It should be noted that matching the instance category and instance quantity of each historical frame image with the instance category and instance quantity of the current frame image can be understood as finding a signboard with the same instance quantity and instance type as in the current frame image.

[0118] In a feasible implementation manner, step S21 may include steps A2101 to A2103:

[0119] Step A2101, obtain a historical frame image, perform image perception analysis on the historical frame image, and obtain semantic point cloud information.

[0120] It is understandable that the historical frame image can be directly obtained from the storage space for storing the image.

[0121] It should be noted that the image perception analysis of the historical frame images to obtain semantic point cloud information includes: obtaining camera intrinsic parameters and camera extrinsic parameters of the target vehicle; obtaining depth information of the historical frame images based on the camera intrinsic parameters and the camera extrinsic parameters; identifying pixel coordinates of semantic information in the historical frame images; and obtaining semantic point cloud information based on the depth information and the pixel coordinates.

[0122] Among them, it can be understood that the camera intrinsic parameters and camera extrinsic parameters of the target vehicle can be obtained from the vehicle equipment data. For example, each frame image of the vehicle is captured by a vehicle-mounted camera, and the camera intrinsic parameters and camera extrinsic parameters of the vehicle-mounted camera will be pre-recorded in the vehicle equipment information and can be directly obtained.

[0123] It should be noted that the depth information of the historical frame image obtained according to the camera internal parameters and the camera external parameters can be obtained through the AI ​​perception model, and a perception map with semantic information can be obtained. The internal and external parameters of the camera are respectively applied to each frame of the semantic perception pixel map. The specific calculation method is shown in the following formula, so that a point cloud with depth information in the local world system is obtained, while retaining its semantic information, and obtaining some point clouds with semantic information, such as ground direction arrows, stop lines, parking lot information, etc. For details, please refer to the following formula:

[0124]

[0125] in, is the pixel coordinate point with semantic information, Z c Represents the depth information in the camera system. represents the camera internal parameters, Represents the camera extrinsics.

[0126] in,

[0127] Step A2102, performing point cloud filtering on the semantic point cloud information to obtain semantic information.

[0128] It is understandable that performing point cloud filtering on the semantic point cloud information may be performing point cloud filtering processing on the selected ground features with semantic point cloud information.

[0129] It should be understood that the semantic information may be information represented by a point cloud in an image, such as arrows, signboards, slogans, and the like.

[0130] Step A2103, performing instance segmentation based on the semantic information to obtain the instance category and instance quantity of the historical frame image.

[0131] It is understandable that each frame of image may contain multiple semantic information, and the semantic information may be the same or different. Instance segmentation based on different semantic information may be to separate the information representing different semantic information as instances with semantic information.

[0132] It should be noted that the semantic information can be segmented through unsupervised learning methods, specifically, mixed Gaussian models, principal component analysis, association rules, deep learning and other segmentation methods. This embodiment does not limit this and can be selected according to actual needs.

[0133] It should be understood that the instance category may be an arrow category, a sign category, and the like.

[0134] In this embodiment, by performing image perception on historical frame images in advance and extracting the instance type and instance quantity of point cloud information in the historical frame images, it is possible to quickly match the instance type and instance quantity of the current frame image with the historical frame images when the current frame image meets the spatiotemporal conditions for loop closure detection, thereby achieving fast and accurate loop closure detection.

[0135] The above are only feasible implementations of step S21 provided in this embodiment, and this embodiment does not specifically limit the specific implementation of step S21.

[0136] Step S22: taking the successfully matched historical frame image as a reference frame image.

[0137] It is understandable that a successful match can be understood as the current frame image having the same instance type as a certain frame reference image, and the number of instances under each instance type is also the same, and it is considered that the image may belong to the same position as the current frame image.

[0138] It should be understood that there may be one or more reference frame images.

[0139] This embodiment provides a vehicle loop detection method, which greatly improves the efficiency of the detection algorithm by limiting the scope of loop detection search and matching, adopts a matching method based on semantic information to detect loop frames, improves the use of traditional ICP algorithms, and adopts ICP algorithm matching based on semantic information to improve matching and loop detection accuracy. In addition, this algorithm is not affected by information such as lighting and has good robustness.

[0140] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the vehicle loop detection method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0141] This application also provides a vehicle loop detection device, please refer to Figure 4 , the vehicle loop detection device comprises:

[0142] An image acquisition module 10 is used to acquire a current frame image captured by a target vehicle;

[0143] An image screening module 20, configured to pre-screen historical frame images based on the current frame image to obtain a reference frame image when the target vehicle meets a preset time and space condition;

[0144] The point cloud matching module 30 is used to perform semantic element matching between the current frame image and the reference frame image to obtain a semantic matching result;

[0145] The loop detection module 40 is used to complete the loop detection of the target vehicle based on the semantic matching result.

[0146] The vehicle loop detection device provided by the present application adopts the vehicle loop detection method in the above embodiment, which can solve the technical problem that the prior art cannot perform vehicle loop detection in real time and accurately. Compared with the prior art, the beneficial effects of the vehicle loop detection device provided by the present application are the same as the beneficial effects of the vehicle loop detection method provided by the above embodiment, and the other technical features in the vehicle loop detection device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0147] The present application provides a vehicle loop detection device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the vehicle loop detection method in the above-mentioned embodiment 1.

[0148] Reference below Figure 5 , which shows a schematic diagram of the structure of a vehicle loop detection device suitable for implementing the embodiment of the present application. The vehicle loop detection device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The vehicle loop detection device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0149] like Figure 5As shown, the vehicle loop detection device may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the vehicle loop detection device are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the vehicle loop detection device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a vehicle loop detection device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or provided instead.

[0150] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0151] The vehicle loop detection device provided by the present application adopts the vehicle loop detection method in the above embodiment, which can solve the technical problem that the prior art cannot perform vehicle loop detection in real time and accurately. Compared with the prior art, the beneficial effects of the vehicle loop detection device provided by the present application are the same as the beneficial effects of the vehicle loop detection method provided by the above embodiment, and the other technical features in the vehicle loop detection device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0152] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0153] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0154] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the vehicle loop detection method in the above-mentioned embodiment.

[0155] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0156] The computer-readable storage medium may be included in the vehicle loop detection device; or may exist independently without being assembled into the vehicle loop detection device.

[0157] The above-mentioned computer-readable storage medium carries one or more programs. When the above-mentioned one or more programs are executed by the vehicle loop detection device, the vehicle loop detection device: obtains a current frame image collected by the target vehicle; when the target vehicle meets the preset time and space conditions, pre-screens the historical frame images based on the current frame image to obtain a reference frame image; matches the semantic elements of the current frame image and the reference frame image to obtain a semantic matching result; and completes loop detection of the target vehicle based on the semantic matching result.

[0158] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0159] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0160] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0161] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned vehicle loop detection method, and can solve the technical problem that the prior art cannot perform vehicle loop detection in real time and accurately. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the vehicle loop detection method provided by the above-mentioned embodiment, and will not be repeated here.

[0162] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned vehicle loop detection method when executed by a processor.

[0163] The computer program product provided by the present application can solve the technical problem that the prior art cannot accurately perform vehicle loopback detection in real time. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the vehicle loopback detection method provided by the above embodiment, which will not be repeated here.

[0164] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A vehicle loop detection method, characterized in that: The vehicle loop detection method comprises: Get the current frame image collected by the target vehicle; When the target vehicle meets the preset time and space conditions, pre-screening the historical frame images based on the current frame image to obtain a reference frame image; Performing semantic element matching between the current frame image and the reference frame image to obtain a semantic matching result; Complete loop detection on the target vehicle based on the semantic matching result.

2. The vehicle loop detection method according to claim 1, characterized in that: When the target vehicle meets the preset time and space conditions, the historical frame images are pre-screened based on the current frame image to obtain the reference frame image, including: Acquire driving parameters of the target vehicle, and obtain driving distance and driving time according to the driving parameters; When the driving distance is greater than or equal to the distance threshold and the driving time is greater than or equal to the time threshold, the target vehicle meets the preset time and space conditions; When the driving distance is less than the distance threshold or the driving duration is less than the duration threshold, the target vehicle does not meet the preset time and space conditions.

3. The vehicle loop detection method according to claim 2, characterized in that: When the driving distance is greater than or equal to the distance threshold and the driving time is greater than or equal to the time threshold, before the target vehicle meets the preset time and space conditions, the method further includes: Obtaining a reference area of ​​the garage where the target vehicle is located, and obtaining a distance threshold according to the reference area; A duration threshold is obtained according to the driving speed parameter of the target vehicle and the reference area.

4. The vehicle loop detection method according to claim 1, characterized in that: When the target vehicle meets the preset time and space conditions, pre-screening the historical frame images based on the current frame image to obtain the reference frame image includes: When the target vehicle meets the preset time and space conditions, the instance category and instance quantity of each historical frame image are matched with the instance category and instance quantity of the current frame image; The successfully matched historical frame image is used as the reference frame image.

5. The vehicle loop detection method according to claim 4, characterized in that: When the target vehicle meets the preset time and space conditions, the historical frame images are pre-screened based on the current frame image to obtain the reference frame image, and the method further includes: Acquire historical frame images, perform image perception analysis on the historical frame images, and obtain semantic point cloud information; Performing point cloud filtering on the semantic point cloud information to obtain semantic information; Instance segmentation is performed based on the semantic information to obtain instance categories and instance quantities of the historical frame image.

6. The vehicle loop detection method according to claim 5, characterized in that: The performing image perception analysis on the historical frame image to obtain semantic point cloud information includes: Obtaining camera intrinsic parameters and camera extrinsic parameters of the target vehicle; Obtaining depth information of the historical frame image according to the camera intrinsic parameters and the camera extrinsic parameters; Identifying pixel coordinates of semantic information in the historical frame image; Semantic point cloud information is obtained according to the depth information and the pixel coordinates.

7. The vehicle loop detection method according to claim 1, characterized in that: The performing semantic element matching between the current frame image and the reference frame image to obtain a semantic matching result includes: Matching point clouds with the same semantic information in the current frame image and the reference frame image to obtain a plurality of point cloud rotation matrices and point cloud translation matrices; Performing consistency evaluation on the multiple point cloud rotation matrices and the point cloud translation matrix to obtain a comprehensive score; A semantic matching result is obtained based on the comprehensive score.

8. A vehicle loop detection device, characterized in that: The vehicle loop detection device comprises: An image acquisition module is used to acquire the current frame image collected by the target vehicle; An image screening module, used for pre-screening historical frame images based on the current frame image to obtain a reference frame image when the target vehicle meets the preset time and space conditions; A point cloud matching module, used for performing semantic element matching between the current frame image and the reference frame image to obtain a semantic matching result; The loop detection module is used to complete the loop detection of the target vehicle based on the semantic matching result.

9. A vehicle loop detection device, characterized in that: The device comprises: a memory, a processor, and a vehicle loop detection program stored in the memory and executable on the processor, wherein the vehicle loop detection program is configured to implement the vehicle loop detection method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores a vehicle loop detection program, and when the vehicle loop detection program is executed by the processor, the vehicle loop detection method according to any one of claims 1 to 7 is implemented.

Images