A loop-back detection method based on text semantics and related equipment

Abstract

The application relates to a loop detection method based on text semantics, which comprises the following steps: acquiring multi-frame three-dimensional point cloud data collected by a laser radar; estimating a mileage counter track by using the multi-frame three-dimensional point cloud data, and screening the multi-frame three-dimensional point cloud according to the mileage counter track to obtain at least one frame of key frame point cloud; acquiring multi-frame two-dimensional image data collected by a camera; identifying the multi-frame two-dimensional image based on a preset text semantic recognition model to obtain multiple text semantics; selecting a key image corresponding to the key frame point cloud and corresponding key text semantics from the multi-frame two-dimensional image data and the corresponding text semantics; matching the key text semantics with historical text semantics corresponding to a historical two-dimensional image set to obtain a candidate loop image; and performing geometric constraint on the key frame point cloud and historical point cloud corresponding to the candidate loop image to obtain a loop detection result. The technical problem of loop detection error caused by a texture similar environment is solved by recognizing text semantics, and the precision and robustness of system positioning and mapping are improved.

Description

Technical Field

[0001] The present application belongs to the field of computer vision technology and relates to a loop detection method based on text semantics and related equipment. Background Art

[0002] Currently, technologies such as autonomous driving, intelligent robots, metaverse, and digital twins are developing rapidly. One of the most important basic technologies among these technologies is positioning technology, which aims to enable an intelligent device to determine its position and posture in the environment, thereby achieving further navigation, control, and mapping.

[0003] The key to achieving positioning is the perception and recognition of the environment. Current technical solutions mainly use cameras to capture images of the surrounding environment, and then perform image recognition to determine the position based on the similarity of the images; or use lidar to scan the surrounding environment to obtain a three-dimensional lidar point cloud, and then determine the position based on the similarity of the point cloud structure. However, these methods have defects. In environments with similar textures or structures, loop detection errors may occur, resulting in positioning errors. Moreover, in low-light environments such as low light intensity or at night, traditional camera-based image recognition methods cannot detect effective information, which leads to positioning failure. In view of the above situation, a loop detection method is urgently needed to solve the problem of ineffective positioning in similar environments or low-light environments.

[0004] The contents of the background technology section are merely publicly known technologies and do not necessarily represent the existing technologies in this field. Summary of the Invention

[0005] The purpose of the present application is to at least to some extent solve the deficiencies in the prior art. In the first aspect, the present application provides a loop detection method based on text semantics, comprising: obtaining multi-frame three-dimensional point cloud data acquired by a lidar, estimating the odometer trajectory using the multi-frame three-dimensional point cloud data and screening the multi-frame three-dimensional point cloud according to the odometer trajectory to obtain at least one frame of key frame point cloud; obtaining multi-frame two-dimensional image data acquired by a camera, recognizing the multi-frame two-dimensional images based on a preset text semantic recognition model to obtain multiple text semantics; selecting key images and corresponding key text semantics corresponding to the key frame point cloud from the multi-frame two-dimensional image data and the corresponding text semantics, matching the key text semantics with the historical text semantics corresponding to the historical two-dimensional image set to obtain candidate loop images; wherein the candidate loop images include the key image and at least one frame of historical two-dimensional image; geometrically constraining the key frame point cloud and the historical point cloud corresponding to the candidate loop image to obtain a loop detection result.

[0006] The second aspect of the present application provides a loop detection device based on text semantics, including: an acquisition unit, used to acquire multi-frame three-dimensional point cloud data collected by a lidar, use the multi-frame three-dimensional point cloud data to estimate the odometer trajectory and filter the multi-frame three-dimensional point cloud according to the odometer trajectory to obtain at least one frame of key frame point cloud; an identification unit, used to acquire multi-frame two-dimensional image data collected by a camera, and recognize the multi-frame two-dimensional images based on a preset text semantic recognition model to obtain multiple text semantics; a matching unit, used to select key images and corresponding key text semantics corresponding to the key frame point cloud from the multi-frame two-dimensional image data and the corresponding text semantics, and match the key text semantics with the historical text semantics corresponding to the historical two-dimensional image set to obtain candidate loop images; wherein the candidate loop images include the key image and at least one frame of historical two-dimensional image; the detection unit, used to geometrically constrain the key frame point cloud and the historical point cloud corresponding to the candidate loop image to obtain a loop detection result.

[0007] The third aspect of the present application provides an intelligent device, including: a laser radar, a camera, and a processor; the laser radar is used to collect multi-frame three-dimensional point cloud data of the surrounding environment of the intelligent device; the camera is used to synchronously collect multi-frame two-dimensional images; the processor is used to obtain multi-frame three-dimensional point clouds and multi-frame two-dimensional images, and process the multi-frame three-dimensional point clouds and multi-frame two-dimensional images according to the text semantics-based loop detection method described in the first aspect of the present application to obtain loop detection results.

[0008] In a fourth aspect, the present application provides a computer-readable storage medium storing a computer program, which is executed by a processor to implement the loop detection method based on text semantics described in the first aspect.

[0009] In a fifth aspect, the present application provides a computer program product. When the computer program code or instructions are executed on a computer, the computer executes the loop detection method based on text semantics described in the first aspect.

[0010] It can be seen from the above-mentioned embodiments of the present application that the present application overcomes the problem that other SLAM methods based on point clouds or images are prone to loop detection errors in environments with similar structures and textures by identifying the text semantics in the scene and calculating the similarity between the identified text semantics and the text semantics corresponding to the historically collected images. It further uses the spatial position information corresponding to the text semantics for re-verification to determine whether similar texts are in the same posture, so that only images with similar text semantics and close positions are confirmed to constitute a loop, thereby solving the problem that other methods using text semantics are not applicable in environments with the same text, and improving the accuracy, efficiency and robustness of system positioning and mapping. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0012] Figure 1 A schematic diagram of a smart device provided in an embodiment of the present application;

[0013] Figure 2 A flowchart of a loop closure detection method based on text semantics provided in an embodiment of the present application;

[0014] Figure 3 A schematic block diagram of a loop closure detection device based on text semantics provided in an embodiment of the present application;

[0015] Figure 4 A schematic block diagram of a computer device provided in an embodiment of the present application;

[0016] Figure 5 A schematic block diagram of a computer-readable storage medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0017] The embodiments of the present application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be understood as limiting the present application. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application without making any creative efforts shall fall within the scope of protection of the present application.

[0018] Figure 1 1 is a schematic diagram of a smart device 100 proposed in an embodiment of the present application. The smart device 100 includes a perception system 110 and a computing platform 120. The perception system 110 includes several sensors for sensing environmental information around the smart device 100, and the computing platform 120 processes images captured by traditional cameras using a classic visual SLAM algorithm to achieve positioning and mapping.

[0019] In the prior art, the perception system 110 is generally a camera, and the classic visual SLAM algorithm specifically includes the following steps: acquiring multiple frames of image data collected by the camera at different times, estimating the pose of the camera at different times in the intelligent device coordinate system according to the image data and establishing a map (i.e., "front end"), and simultaneously, calculating the similarity between images to obtain loop detection information; optimizing the pose and map of the intelligent device according to the camera pose, the map and the loop detection information at different times to obtain a globally consistent trajectory and map (i.e., "back end"); and establishing a map required by the task setting according to the trajectory.

[0020] However, the image method based on the traditional camera cannot extract effective information in a weak light or dark environment, and may cause loop detection errors in an environment with similar textures or structures, thereby causing positioning failure. To this end, the laser radar and the camera are used as the perception system 110 in the present application, and the loop detection method based on text semantics provided by one or more embodiments of the present application is used by the computing platform 120 to solve the above problems, so that the intelligent device can realize effective positioning in a similar environment or a weak light environment.

[0021] In one embodiment, the perception system 110 includes a laser radar and a camera, the laser radar is used to sense the three-dimensional point cloud of the environment around the intelligent device 100, and the camera is used to synchronously sense the image of the environment around the intelligent device 100. The camera can be an infrared camera, a grayscale camera, a color camera or a depth camera, etc. The present application is beneficial to realize accurate positioning in a weak light environment and in a similar environment by performing semantic extraction on the image collected by the camera and combining the odometer estimation provided by the laser radar point cloud. It should be noted that the camera is preferably an infrared camera, because its passive light sensing characteristic makes it less affected by factors such as light, and it is more suitable for use in dark or weak light environments, thereby realizing more accurate positioning.

[0022] The computing platform 120 can include a processor and a memory, wherein the processor is a circuit having a signal processing capability, and the memory is used to store the program instructions corresponding to the loop detection method based on text semantics provided by one or more embodiments of the present application. Part or all of the processors in the processor can call the instructions in the memory to acquire the three-dimensional point cloud collected by the laser radar and the two-dimensional image collected by the camera, and process the three-dimensional point cloud and the two-dimensional image by the loop detection method based on text semantics provided by one or more embodiments of the present application to obtain the loop detection result.

[0023] In an embodiment, the processor can be a circuit with instruction reading and running capability, such as a central processing unit (CPU), a microprocessor, a graphics processing unit (GPU) (which can be understood as a kind of microprocessor), a digital signal processor (DSP), or the like; in another implementation, the processor can realize certain functions through a logical relationship of hardware circuit, which is fixed or can be reconfigured, such as an application-specific integrated circuit (ASIC) or a programmable logic device (PLD) implemented hardware circuit, such as an FPGA. In the reconfigurable hardware circuit, the processor loads the configuration document to realize the hardware circuit configuration process, which can be understood as the process of the processor loading instructions to realize the functions of the above part or all units. In addition, the processor can also be a hardware circuit designed for artificial intelligence, which can be understood as a kind of ASIC, such as a neural network processing unit (NPU), a tensor processing unit (TPU), a deep learning processing unit (DPU), etc.

[0024] The intelligent device 100 in the present application can include: a road vehicle, a water vehicle, an air vehicle, an industrial device, an agricultural device, a wearable device, a mobile robot, or an entertainment device, etc. For example, the intelligent device can be a vehicle, which is a general concept of a vehicle, and can be a vehicle (such as a commercial vehicle, a passenger vehicle, a motorcycle, a flying vehicle, a train, etc.), an industrial vehicle (such as a forklift, a trailer, a tractor, etc.), an engineering vehicle (such as an excavator, a bulldozer, a crane, etc.), an agricultural device (such as a mower, a harvester, etc.), an amusement device, a toy vehicle, etc. The type of vehicle is not limited in the embodiments of the present application.

[0025] Figure 2 is a flowchart of a text semantic-based loop detection method provided by the embodiments of the present application, which is suitable for an environment with text elements, and specifically includes:

[0026] S210: Obtain a plurality of frames of three-dimensional point clouds collected by a laser radar, estimate a mileage counter track by using the plurality of frames of three-dimensional point cloud data, and filter the plurality of frames of three-dimensional point clouds according to the mileage counter track to obtain at least one frame of key frame point cloud.

[0027] Specifically, the laser radar is used to scan the surrounding environment of the smart device 100 to obtain multiple frames of three-dimensional point clouds, and the camera is used to collect multiple frames of two-dimensional images of the same surrounding environment while the laser radar is operating. The laser radar and the camera share the system timestamp of the smart device, and the relative positions of the laser radar and the camera are fixed during operation. It should be noted that, given that the relative positions of the laser radar and the camera are fixed during operation, the relative external parameters between the two can be calibrated using a calibration algorithm before use, so that at the same timestamp, at least the point cloud data collected by the laser radar and some pixels of the two-dimensional image collected by the camera have a one-to-one correspondence, that is, a certain point in the point cloud data collected by the laser radar can find a corresponding point on the two-dimensional image collected by the camera, and vice versa.

[0028] Furthermore, given the huge amount of multi-frame 3D point cloud data and the possible presence of flying points, directly using the multi-frame 3D point cloud for processing will result in large computational complexity and low accuracy. Therefore, before using the multi-frame 3D point cloud to estimate the odometry trajectory, the multi-frame 3D point cloud can also be subjected to necessary preprocessing, such as filtering to remove noise (such as flying points), downsampling to reduce the data volume, etc., so as to efficiently obtain a more accurate odometry trajectory.

[0029] In one embodiment, estimating the odometer trajectory using multiple frames of three-dimensional point clouds includes: extracting features from multiple frames of three-dimensional point clouds to obtain landmark features, such as landmark features such as corner points, plane points, or edge points; using the same landmark features between different frame point clouds to estimate the relative poses between each frame point cloud, including rotation and / or translation, and calculating the cumulative pose of each frame point cloud relative to the starting frame point cloud based on the relative poses between each frame point cloud to obtain the odometer trajectory. It should be noted that the odometer trajectory is composed of the cumulative poses of each frame point cloud relative to the starting frame point cloud and the timestamps corresponding to each frame point cloud. The starting frame point cloud can be the first frame of three-dimensional point cloud collected by the lidar at the initial moment, or it can be the three-dimensional point cloud corresponding to the earlier timestamp in the multiple frame three-dimensional point clouds. In actual application, it can be selected according to the actual situation, and this application does not impose any restrictions on this.

[0030] In one embodiment, filtering multiple frames of three-dimensional point clouds according to the odometer trajectory to obtain key frame point clouds includes: taking the starting frame point cloud as the starting point, obtaining the motion state of the smart device based on the odometer trajectory, and extracting a frame of three-dimensional point cloud as the key frame point cloud every time the motion state of the smart device exceeds a preset distance threshold and / or a preset angle threshold, thereby reducing the loop detection frequency and improving the loop detection efficiency. Specifically, the starting frame point cloud is taken as the first key frame point cloud, and the motion state of the smart device is obtained based on the odometer trajectory, that is, the distance change values ​​and / or angle change values ​​corresponding to the remaining frame three-dimensional point clouds and the starting frame point cloud are calculated. If the distance change value and / or angle change value between the point clouds between frames exceeds the preset distance threshold and / or preset angle threshold, the three-dimensional point cloud with a timestamp close to the starting frame point cloud is extracted as the second frame key frame point cloud, and then the second frame key frame point cloud is used as a reference to calculate the distance change values ​​and / or angle change values ​​of the three-dimensional point clouds corresponding to the remaining time stamps. If there is a distance change value and / or angle change value of the point cloud between frames that exceeds the threshold, the three-dimensional point cloud with a timestamp close to the second frame key frame point cloud is extracted as the third frame key frame point cloud, and the remaining time stamp point clouds are extracted by analogy. Key frame point clouds will not be repeated here.

[0031] S220: Acquire multiple frames of two-dimensional image data captured by a camera, and recognize the multiple frames of two-dimensional image data based on a preset text semantic recognition model to obtain multiple text semantics.

[0032] In some embodiments, recognizing multiple frames of two-dimensional images based on a preset text semantic recognition model to obtain multiple text semantics includes: obtaining high-quality two-dimensional images from the multiple frames of two-dimensional images, and using the preset text semantic recognition model to perform text recognition on the multiple frames of high-quality two-dimensional images to obtain multiple text semantics.

[0033] In one embodiment, obtaining a high-quality two-dimensional image from multiple frames of two-dimensional images includes: deleting low-quality images such as low image resolution, low contrast, overexposure or underexposure from the multiple frames of two-dimensional images, and retaining only high-quality images for processing, which not only improves the efficiency of loop detection, but also ensures the accuracy of loop detection.

[0034] In another embodiment, obtaining high-quality two-dimensional images from multiple frames of two-dimensional images includes: preprocessing the multiple frames of two-dimensional images to obtain multiple frames of high-quality two-dimensional images, specifically, interpolating the multiple frames of two-dimensional images, and / or performing brightness or contrast on the multiple frames of two-dimensional images, and / or filtering the multiple frames of two-dimensional images, and / or performing edge detection on the multiple frames of two-dimensional images to enhance edge-highlighted text areas, thereby improving the image quality of the multiple frames of two-dimensional images to obtain multiple frames of high-quality two-dimensional images, so as to improve the accuracy of loop detection.

[0035] In one embodiment, a preset text semantic recognition model includes: a detection module, a positioning module, an extraction module, and a recognition module. Using the preset text semantic recognition model to perform text recognition on multiple frames of high-quality two-dimensional images to obtain multiple text semantics includes: inputting the multiple frames of high-quality two-dimensional images one by one into the detection module in the preset text semantic recognition model, so as to identify the area where the text is located in the multiple frames of high-quality two-dimensional images using a text detection algorithm to obtain the text area. The text detection algorithm is preferably a detection algorithm based on deep learning, such as EAST, FOTS, CTPN, etc. The positioning module in the preset text semantic recognition model is used to locate the position information of the text area in the multiple frames of high-quality two-dimensional images. According to the position information of the text area, the text area corresponding to the multiple frames of high-quality two-dimensional images is cropped using the extraction module in the preset text semantic recognition model to obtain a text image that only includes the text area; and the recognition module is used to perform text recognition and semantic analysis on the text image that only includes the text area corresponding to the multiple frames of high-quality two-dimensional images to obtain multiple text semantics.

[0036] Preferably, using a recognition module to perform text recognition and semantic analysis on text images that only include text areas corresponding to multiple frames of high-quality two-dimensional images to obtain multiple semantic texts includes: using OCR technology (such as Tesseract, Google Vision API, etc.) to recognize text in text images that only include text areas corresponding to multiple frames of high-quality two-dimensional images, and performing semantic analysis on the recognized text through natural language processing (NLP) technology to understand its meaning to obtain multiple text semantics corresponding to the multiple frames of high-quality two-dimensional images, wherein the text semantics include character strings such as words, numbers, and symbols.

[0037] After obtaining multiple text semantics corresponding to multiple frames of two-dimensional images, it also includes: comparing the text semantics corresponding to multiple consecutive frames of two-dimensional images. If the text semantics between two frames of images are the same, any one of the frames of images is deleted and the text corresponding to the other frame of image is retained to reduce the amount of calculation and improve the efficiency of loop detection.

[0038] It should be noted that the preset text semantic recognition model is a pre-trained neural network model. The text semantic recognition model to be trained can be trained by using images including text and images without text as sample sets, and the training results can be constrained by a loss function to obtain the preset text semantic recognition model of this application.

[0039] S230: Select a key image and corresponding key text semantics corresponding to the key frame point cloud from multiple frames of two-dimensional image data and corresponding text semantics, and match the key text semantics with the historical text semantics corresponding to the historical two-dimensional image set to obtain a candidate loop image; wherein the candidate loop image includes the key image and at least one frame of historical two-dimensional image.

[0040] Specifically, while controlling the lidar to collect multiple frames of three-dimensional point cloud data, the camera is synchronously controlled to collect multiple frames of two-dimensional image data. The acquisition frequency of the lidar is greater than the acquisition frequency of the camera. Therefore, when low-quality images in multiple frames of two-dimensional images are deleted to obtain high-quality images for text semantic recognition, the remaining two-dimensional images still have images with the same timestamp or the nearest timestamp corresponding to the key frame point cloud.

[0041] In one embodiment, an image and its text semantics corresponding to a timestamp of a key frame point cloud are selected from multi-frame two-dimensional image data and corresponding text semantics as a key image and corresponding key text semantics, including: if an image with the same timestamp as the key frame point cloud exists in the multi-frame two-dimensional image, the image and its text semantics with the same timestamp are selected as the key image and corresponding key text semantics; if low-quality images are deleted from the multi-frame two-dimensional image so that there is no image with the same timestamp as the key frame point cloud in the multi-frame two-dimensional image, the image and text semantics with the closest timestamp to the key frame point cloud are selected as the key image and corresponding text semantics.

[0042] Specifically, according to the timestamp corresponding to the key frame point cloud, the image with the same timestamp or the closest timestamp is selected from multiple frames of two-dimensional images to obtain the key image, and then the corresponding key text semantics are obtained according to the key image, and the key text semantics are matched with the historical text semantics corresponding to the historical two-dimensional image set to obtain the candidate loop image; wherein, the historical two-dimensional image set is composed of two-dimensional images collected at historical moments, the historical moments are the moments before the timestamp corresponding to the key frame image, and the historical text semantics are the text semantics corresponding to the two-dimensional images collected at the historical moments.

[0043] In one embodiment, matching the key text semantics with the historical text semantics corresponding to each historical two-dimensional image in the historical two-dimensional image set to obtain a candidate loop image includes: calculating the similarity between the key text semantics corresponding to the selected key image and the historical text semantics corresponding to each historical two-dimensional image in the historical two-dimensional image set; if the similarity between the selected key image and a certain historical two-dimensional image exceeds a preset similarity threshold, it means that the selected key image and the historical two-dimensional image have formed a preliminary loop, and the two frames of images are regarded as candidate loop images; if it is lower than the preset similarity threshold, it means that no loop is formed, and processing continues with the next key image.

[0044] However, as a ubiquitous element in man-made environments, text often contains the same text, which can easily lead to loop detection errors. Moreover, since the acquisition of text images is easily affected by factors such as changes in lighting conditions and robot speed, the same text may have different recognition effects each time it is acquired. For example, the text is actually the same but the recognition results are different, or different texts are recognized as the same, which in turn affects the accuracy of loop detection. To this end, this application proposes a matching method based on spatial text semantics for loop detection, thereby improving the accuracy of loop detection.

[0045] In some embodiments, the matching method based on spatial text semantics includes: calculating the similarity between the text semantics corresponding to the selected key image and the historical text semantics corresponding to each historical two-dimensional image in the historical two-dimensional image set; obtaining the spatial position information with the same or closest timestamp as the selected key image from the odometer trajectory obtained based on the multi-frame three-dimensional point cloud, and performing a distance calculation on it with the historical spatial position information corresponding to the historical two-dimensional image set to obtain the spatial distance between the text semantics corresponding to the selected key image and the historical text semantics corresponding to each historical two-dimensional image in the historical two-dimensional image set; determining the candidate loop image based on the similarity and spatial distance between the selected key image and each historical image in the historical two-dimensional image set.

[0046] Furthermore, spatial text semantic matching is represented as a composite function ST, specifically:

[0047]

[0048] Where i represents the selected key image of frame i, j represents the historical two-dimensional image of frame j in the historical two-dimensional image set; S ij represents the similarity of the textual semantics between the key image of the i-th frame and the historical two-dimensional image of the j-th frame in the historical two-dimensional image set; |S i |,|S j | represents the string corresponding to the text semantics of the key image of the i-th frame and the j-th historical two-dimensional image in the historical two-dimensional image set; d(S i , S j ) represents the conversion distance between the string corresponding to the text semantics of the i-th key image and the string corresponding to the text semantics of the j-th historical two-dimensional image in the historical two-dimensional image set, and represents the minimum operations required to convert the string corresponding to the text semantics of the i-th key image into the string corresponding to the text semantics of the j-th historical two-dimensional image in the historical two-dimensional image set, where the operations include one or more combinations of deletion, insertion, and substitution; T ij represents the spatial distance between the key image of frame i and the historical two-dimensional image of frame j in the historical two-dimensional image set, (T ix , T iy , T iz) is represented as the spatial position information of the key image of the i-th frame, which can be obtained from the odometer trajectory. (T jx , T jy , T jz ) represents the historical spatial position information corresponding to the j-th historical two-dimensional image in the historical two-dimensional image set.

[0049] In one embodiment, determining whether a selected key image is a candidate loop image based on the similarity and spatial distance between the selected key image and each historical image in the historical two-dimensional image set includes: comparing the similarity and spatial distance between the selected key image and each historical image in the historical two-dimensional image set with a preset similarity threshold and a preset spatial distance threshold, respectively, to determine a candidate loop image; wherein, when the similarity between the selected key image and a historical image in the historical two-dimensional image set is greater than the preset similarity threshold and the spatial distance is less than the preset spatial threshold, the selected key image and the historical two-dimensional image in the historical two-dimensional image set constitute a preliminary loop, and the selected key image and the corresponding historical two-dimensional image in the historical two-dimensional image set are regarded as candidate loop images; in other cases, no preliminary loop is constituted, and the next frame of the key image will continue to be processed.

[0050] In another embodiment, determining a candidate loop image based on the similarity and spatial distance between the selected key image and each historical image in the historical two-dimensional image set includes: adding or weighted adding the similarity and spatial distance between the selected key image and each historical image in the historical two-dimensional image set, and comparing with a preset candidate threshold; if the addition result of the selected key image and a historical two-dimensional image in the historical two-dimensional image set is within the preset candidate threshold, an initial loop is constituted, and the above two frames of images are determined to constitute a preliminary loop and are determined to be candidate loop images; otherwise, an initial loop is not constituted, and processing of the next key image frame continues.

[0051] In order to prevent the distance constituting the initial loop from being too short, the present application also calculates the time interval between the two frames of images based on the timestamps of the key image corresponding to the initial loop and the corresponding historical two-dimensional image in the historical two-dimensional image set. If the time interval between the two frames of images is less than or equal to the preset time threshold range, the loop is abandoned and the next frame of image is processed; otherwise, it is confirmed that the initial loop is formed, thereby eliminating the situation where the smart device continuously captures the same scene in a short period of time, so as to improve the loop accuracy.

[0052] It should be noted that in addition to the above methods, the i-th frame key image can also be matched with the text semantics and spatial position information of the j-th frame image in the historical two-dimensional image set through fuzzy matching methods such as regular expressions, approximate string matching algorithms, fuzzy logic, and machine learning. This application does not impose any restrictions on this.

[0053] S240: Perform geometric constraints on the key frame point cloud and historical point cloud corresponding to the candidate loop closure image to obtain a loop closure detection result.

[0054] Specifically, the historical point cloud corresponding to the historical two-dimensional image in the candidate loop image is obtained, and the corresponding key frame point cloud is obtained according to the timestamp of the key frame image in the candidate loop image; the Euclidean distance between the key frame point cloud and the historical point cloud is calculated and the loop detection result is determined based on the Euclidean distance; wherein, if the obtained distance value is greater than the preset Euclidean distance value, the loop detection result indicates that the candidate loop image does not constitute a loop, otherwise it constitutes a loop.

[0055] After determining that the selected key image and the historical two-dimensional image form a loop, the loop detection results can also be used to optimize the odometer trajectory and historical map to obtain a high-precision global map and trajectory; among them, the historical map is a map obtained by splicing historical two-dimensional images collected at historical moments.

[0056] In summary, the present application overcomes the problem that other SLAM methods based on point clouds or images are prone to loop detection errors in environments with similar structures and textures by identifying the text semantics in the scene and calculating the similarity between the identified text semantics and the text semantics corresponding to the historically collected images. It further uses the spatial position information corresponding to the text semantics for re-verification to determine whether similar texts are in the same posture, so that only images with similar text semantics and close positions are confirmed to constitute loops, thereby solving the problem that other methods using text semantics are not applicable in environments with the same text, and improving the accuracy, efficiency and robustness of system positioning and mapping.

[0057] Figure 3 This is a schematic block diagram of a positioning device 300 for fusing 3D lidar point clouds and infrared images, provided in an embodiment of the present application. The device 300 includes: an acquisition unit 310 for acquiring multiple frames of 3D point cloud data acquired by a lidar, estimating an odometer trajectory using the multiple frames of 3D point cloud data, and filtering the multiple frames of 3D point cloud data based on the odometer trajectory to obtain at least one keyframe point cloud; a recognition unit 320 for acquiring multiple frames of 2D image data acquired by a camera, recognizing the multiple frames of 2D images based on a preset text semantic recognition model to obtain multiple text semantics; a matching unit 330 for selecting a key image and a key text semantic corresponding to the keyframe point cloud from the multiple frames of 2D image data and the corresponding text semantics, and matching the key text semantics with the historical text semantics corresponding to the historical 2D image set to obtain a candidate loop image; wherein the candidate loop image includes the key image and at least one frame of historical 2D image; and a detection unit 340 for geometrically constraining the keyframe point cloud and the historical point cloud corresponding to the candidate loop image to obtain a loop detection result.

[0058] The detailed description of the device 300 proposed in the present application can refer to the foregoing embodiments, and the same content will not be repeated here. In some possible embodiments, the processor in the smart device 100 described above can include the device 300 described above.

[0059] Figure 4 is a schematic block diagram of a computer device 400 provided in an embodiment of the present application. Figure 4 The computer device 400 shown includes a memory 410, a processor 420, and a bus 440. Optionally, the computer device 400 further includes a communication interface 430. The memory 410, the processor 420, and the communication interface 430 are communicatively connected to each other through the bus 440.

[0060] The memory 410 can be a read only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 410 can store a program, and when the program stored in the memory 410 is executed by the processor 420, the processor 420 is configured to perform various steps of the method proposed in the embodiments of the present application.

[0061] The processor 420 can be a general-purpose central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU), or one or more integrated circuits, configured to execute related programs to implement the method proposed in the embodiments of the present application.

[0062] The processor 420 can also be an integrated circuit chip with a signal processing capability, and in the implementation process, various steps of the method described in the embodiments of the present application can be completed through the integrated logic circuit or the instruction in the form of software in the processor 420.

[0063] The above-mentioned processor 420 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The various methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method involved in the embodiments of the present application can be directly reflected as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers, etc. The storage medium is located in the memory 410, and the processor 420 reads the information in the memory 410 and completes it in combination with its hardware. Figure 3 The units included in the device shown need to perform the functions, or perform the methods described in the method embodiments of the present application.

[0064] The communication interface 430 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the apparatus 400 and other devices or a communication network.

[0065] Bus 440 may include a pathway for transmitting information between various components of device 400 (eg, memory 410 , processor 420 , communication interface 430 ).

[0066] It should be understood that although the above-mentioned device 400 only shows a memory, a processor, and a communication interface, in the specific implementation process, those skilled in the art should understand that the device 400 may also include other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the device 400 may also include hardware devices that implement other additional functions. In addition, those skilled in the art should understand that the device 400 may also include only the devices necessary to implement the embodiments of the present application, and does not necessarily include Figure 4 All devices shown in .

[0067] It should be appreciated that the processor in the embodiments of the present application can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic components, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0068] It should also be understood that the memory in the embodiments of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically EPROM (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM) used as an external cache. By way of example but not limitation, many forms of random access memory (RAM) are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM) and direct rambus RAM (DR RAM).

[0069] Figure 5 is a schematic block diagram of a computer readable storage medium 500 provided by the embodiments of the present application. Figure 5The computer-readable storage medium 500 shown in FIG. 5 stores computer instructions 510. When the computer instructions 510 are executed by a processor, the method corresponding to the above embodiment can be implemented.

[0070] In some possible embodiments, the computer-readable storage medium 500 can be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that includes a collection of one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, or magnetic tapes), optical media (e.g., DVDs), or semiconductor media. The semiconductor media can be a solid-state drive.

[0071] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0072] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0073] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0074] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0075] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0076] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0077] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A loop detection method based on text semantics, characterized in that: Applied to smart devices including lidar and cameras, the loop detection method includes: Acquire multiple frames of three-dimensional point cloud data collected by the laser radar, estimate an odometer trajectory using the multiple frames of three-dimensional point cloud data, and filter the multiple frames of three-dimensional point cloud data according to the odometer trajectory to obtain at least one key frame point cloud; Acquire multiple frames of two-dimensional image data captured by the camera, and recognize the multiple frames of two-dimensional image data based on a preset text semantic recognition model to obtain multiple text semantics; Selecting a key image and a corresponding key text semantic corresponding to the key frame point cloud from the multiple frames of two-dimensional image data and the corresponding text semantics, and matching the key text semantics with the historical text semantics corresponding to the historical two-dimensional image set to obtain a candidate loop image; wherein the candidate loop image includes the key image and at least one frame of historical two-dimensional image; The step of matching the key text semantics with the historical text semantics corresponding to the historical two-dimensional image set to obtain a candidate loop image includes: Calculating the similarity between the text semantics corresponding to the selected key image and the historical text semantics corresponding to each historical two-dimensional image in the historical two-dimensional image set; Obtaining spatial position information having the same or closest timestamp as the selected key image from an odometry trajectory obtained based on a multi-frame three-dimensional point cloud, and calculating the distance between the spatial position information obtained from the odometry trajectory and historical spatial position information corresponding to the historical two-dimensional image set, thereby obtaining the spatial distance between the text semantics corresponding to the selected key image and the historical text semantics corresponding to each historical two-dimensional image in the historical two-dimensional image set; Determine a candidate loop image based on the similarity and spatial distance between the selected key image and each historical image in the historical two-dimensional image set; Geometric constraints are performed on the key frame point cloud and the historical point cloud corresponding to the candidate loop closure image to obtain the loop closure detection result.

2. The loop detection method according to claim 1, wherein: Estimating an odometry trajectory using the multi-frame three-dimensional point cloud data, including: Performing feature extraction on the multi-frame three-dimensional point cloud to obtain landmark features; The same landmark features between different frame point clouds are used to estimate the relative poses between each frame point cloud, including rotation and / or translation. The cumulative pose of each frame point cloud relative to the starting frame point cloud is calculated based on the relative poses between the frame point clouds to obtain the odometry trajectory.

3. The loop detection method according to claim 1, wherein: Filtering the multiple frames of three-dimensional point cloud according to the odometer trajectory to obtain at least one key frame point cloud, including: Taking the starting frame point cloud as the starting point, the motion state of the smart device is obtained based on the odometer trajectory. When the motion state of the smart device exceeds a preset distance threshold and / or a preset angle threshold, a frame of 3D point cloud is extracted from the multiple frames of 3D point cloud as a key frame point cloud.

4. The loop detection method according to claim 1, wherein: The method of recognizing multiple frames of two-dimensional images based on a preset text semantic recognition model to obtain multiple text semantics includes: High-quality two-dimensional images are obtained from multiple frames of two-dimensional images, and a preset text semantic recognition model is used to perform text recognition on the multiple frames of high-quality two-dimensional images to obtain multiple text semantics.

5. The loop detection method according to claim 4, wherein: The step of acquiring a high-quality two-dimensional image from a plurality of two-dimensional image frames comprises: Deleting low-quality images corresponding to low image resolution, low contrast, overexposure or underexposure in the multiple frames of two-dimensional images; and / or interpolating the multiple frames of two-dimensional images, and / or adjusting the brightness or contrast of the multiple frames of two-dimensional images, and / or filtering the multiple frames of two-dimensional images, and / or performing edge detection on the multiple frames of two-dimensional images to enhance the edges of prominent text areas, so as to obtain the high-quality images.

6. The loop detection method according to claim 1, wherein: The step of matching the key text semantics with the historical text semantics corresponding to the historical two-dimensional image set to obtain a candidate loop image includes: Calculating the similarity between the key text semantics corresponding to the selected key image and the historical text semantics corresponding to each historical two-dimensional image in the historical two-dimensional image set; If the similarity between the selected key image and a certain historical two-dimensional image in the historical two-dimensional image set exceeds a preset similarity threshold, the selected key image and the corresponding historical two-dimensional image are regarded as candidate loop images; If the similarity is lower than the preset threshold, it does not constitute a loop and continues to process the next key image.

7. The loop detection method according to claim 1, wherein: Geometric constraints are applied to the key frame point cloud and the historical point cloud corresponding to the candidate loop closure image to obtain loop closure detection results including: Obtaining a historical point cloud corresponding to a historical two-dimensional image in the candidate loop closure image, and obtaining a corresponding key frame point cloud according to a timestamp of a key frame image in the candidate loop closure image; Calculate the Euclidean distance between the key frame point cloud and the historical point cloud, and determine the loop detection result based on the Euclidean distance; wherein, if the obtained distance value is greater than the preset Euclidean distance value, the loop detection result indicates that the candidate loop image does not constitute a loop, otherwise it constitutes a loop.

8. A loop detection device based on text semantics, characterized in that: include: an acquisition unit, configured to acquire multiple frames of three-dimensional point cloud data acquired by a laser radar, estimate an odometer trajectory using the multiple frames of three-dimensional point cloud data, and filter the multiple frames of three-dimensional point cloud data according to the odometer trajectory to obtain at least one key frame point cloud; A recognition unit is configured to obtain a plurality of frames of two-dimensional image data captured by a camera, and recognize the plurality of frames of two-dimensional image data based on a preset text semantic recognition model to obtain a plurality of text semantics; a matching unit configured to select a key image and a corresponding key text semantic having the same or closest timestamp as the key frame point cloud from the multiple frames of two-dimensional image data and the corresponding text semantics, and match the key text semantics with the historical text semantics corresponding to the historical two-dimensional image set to obtain a candidate loop image; wherein the candidate loop image includes the key image and at least one frame of historical two-dimensional image; The step of matching the key text semantics with the historical text semantics corresponding to the historical two-dimensional image set to obtain a candidate loop image includes: Calculating the similarity between the text semantics corresponding to the selected key image and the historical text semantics corresponding to each historical two-dimensional image in the historical two-dimensional image set; Obtaining spatial position information having the same or closest timestamp as the selected key image from an odometry trajectory obtained based on a multi-frame three-dimensional point cloud, and calculating the distance between the spatial position information obtained from the odometry trajectory and historical spatial position information corresponding to the historical two-dimensional image set, thereby obtaining the spatial distance between the text semantics corresponding to the selected key image and the historical text semantics corresponding to each historical two-dimensional image in the historical two-dimensional image set; Determine a candidate loop image based on the similarity and spatial distance between the selected key image and each historical image in the historical two-dimensional image set; The detection unit is used to perform geometric constraints on the key frame point cloud and the historical point cloud corresponding to the candidate loop closure image to obtain a loop closure detection result.

9. A smart device, characterized in that: include: A laser radar, used to collect multi-frame three-dimensional point clouds of the surrounding environment of the smart device; A camera, configured to synchronously capture multiple frames of two-dimensional images of the surrounding environment of the smart device; A processor is used to obtain the multi-frame three-dimensional point cloud and the multi-frame two-dimensional image, and process the multi-frame three-dimensional point cloud and the multi-frame two-dimensional image according to the loop closure detection method according to any one of claims 1 to 7 to obtain a loop closure detection result.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the loop detection method according to any one of claims 1 to 7.

Images