A loop detection method based on active vision and text semantics and related equipment

Abstract

The application relates to a loop detection method based on active vision and text semantics, which comprises the following steps: changing the pose of a mechanical arm to drive a camera to collect images of a surrounding environment to obtain front-view text images; synchronously controlling a laser radar to collect multiple frames of point clouds to screen key frame point clouds; selecting key text images corresponding to the key frame point clouds from the multiple frames of front-view text images, extracting text information in the key text images and converting the text information to a global coordinate system; calculating the similarity of the text information in the key text images in the global coordinate system and historical text information obtained at a historical moment, determining candidate loop point clouds according to the similarity; and performing geometric consistency calculation on the candidate loop point clouds to obtain distance values to obtain loop detection results. The active vision is used to collect front-view images of text information, and the double loop verification of the point clouds is combined to solve the technical problems that feature degradation, sparseness or similar environment lead to loop detection errors, and the precision and robustness of system positioning and mapping are improved.

Description

Technical Field

[0001] The present application belongs to the field of computer vision technology and relates to a loop detection method and related equipment based on active vision and text semantics. Background Art

[0002] Currently, technologies such as autonomous driving, intelligent robots, metaverse, and digital twins are developing rapidly. One of the most important basic technologies among these technologies is Simultaneous Localization and Mapping (SLAM) technology. Its purpose is to enable an intelligent device to determine its position and posture in an unknown environment, thereby further realizing autonomous navigation, control, and mapping.

[0003] The key to realizing SLAM technology is the perception and recognition of the environment. Existing technologies usually use LiDAR-based SLAM methods, which mainly rely on the geometric features of point clouds for registration and achieve positioning and map construction by comparing feature points extracted from the environment. However, this type of method performs well in static and simple environments, but is often affected by feature loss or mismatching in dynamic or complex environments, resulting in reduced accuracy.

[0004] With the advancement of technology and increasing demand, SLAM technology has been improved in many aspects, especially through techniques such as semantic SLAM, active view control (AVC), and loop closure detection to improve positioning accuracy. Semantic SLAM improves scene understanding by incorporating semantic information in the environment, such as object categories and attributes. However, this method relies too much on geometric features, resulting in a significant decrease in registration accuracy when environmental features are sparse or degraded, leading to inaccurate or failed positioning. While active view control can select the optimal view angle during map construction to improve the perception of key information, it requires angle-by-angle adjustment to select the optimal angle, which is inefficient. Loop closure detection aims to identify the same location when revisiting it to correct accumulated pose errors and ensure global map consistency. However, in complex or ambiguous scenes, loop closure detection is susceptible to mismatches, leading to map inconsistencies, especially in environments with a large number of similar or repetitive structures. Given this situation, a method is urgently needed to address the problems of localization and mapping in feature-degraded or dynamic environments.

[0005] The contents of the background technology section are merely publicly known technologies and do not necessarily represent the existing technologies in this field. Summary of the Invention

[0006] The purpose of the present application is to at least to some extent solve the deficiencies in the prior art. The first aspect of the present application provides a loop detection method based on active vision and text semantics, including: controlling a robotic arm to drive a camera to capture images of the surrounding environment, and detecting text information on the captured images. If text information is detected, the posture of the end of the robotic arm is changed to move the camera to capture a front-view text image including text information; synchronously controlling a laser radar to capture multiple frames of point clouds, using the multiple frames of point clouds to estimate the odometer trajectory and screening the multiple frames of three-dimensional point clouds according to the odometer trajectory to obtain at least one frame of key frame point cloud; in the multiple frames of front-view A key text image with the same timestamp as the key frame point cloud is selected from the text image, text extraction is performed on the key text image to obtain text information, and the current frame point cloud and text information are converted to the global coordinate system; the similarity between the text information in the key text image in the global coordinate system and the historical text information obtained at the historical moment is calculated, and the candidate loop point cloud is determined according to the similarity. The candidate loop point cloud includes the key frame point cloud and the historical frame point cloud corresponding to the historical text information; the geometric consistency calculation is performed on the key frame point cloud and the historical frame point cloud to obtain the distance value, and the distance value is compared with the preset distance threshold to obtain the loop detection result.

[0007] The second aspect of the present application provides a loop detection device based on active vision and text semantics, which includes: a first control unit for controlling a robotic arm to drive a camera to capture images of the surrounding environment, and to detect text information on the captured images. If text information is detected, the posture of the end of the robotic arm is changed to move the camera to capture a front-view text image including text information; a second control unit for synchronously controlling a laser radar to capture multiple frames of point clouds, using the multiple frames of point clouds to estimate the odometer trajectory and screening the multiple frames of three-dimensional point clouds according to the odometer trajectory to obtain at least one frame of key frame point cloud; a conversion unit for selecting a key frame in the multiple frames of front-view text images. A key text image with the same timestamp as the key frame point cloud is taken, text extraction is performed on the key text image to obtain text information, and the current frame point cloud and text information are converted to the global coordinate system; a calculation unit is used to calculate the similarity between the text information in the key text image in the global coordinate system and the historical text information obtained at the historical moment, and determine the candidate loop point cloud according to the similarity, and the candidate loop point cloud includes the key frame point cloud and the historical frame point cloud corresponding to the historical text information; a detection unit is used to perform geometric consistency calculation on the key frame point cloud and the historical frame point cloud to obtain a distance value, and compare the distance value with the preset distance threshold to obtain a loop detection result.

[0008] The third aspect of the present application provides a positioning system for use on smart devices, including a PID controller robotic arm, a camera, a laser radar and a processing platform, wherein: the laser radar is mounted on the smart device body and is used to sense the three-dimensional point cloud of the surrounding environment of the smart device 100; the camera is installed on the smart device body through the robotic arm, and under the control of the PID controller, the robotic arm can change its own posture and thereby drive the camera to collect image data of the required perspective; the processing platform is used to control the laser radar and the camera to respectively collect multiple frames of point clouds and multiple frames of images and process them according to the loop detection method of the first aspect mentioned above to obtain loop detection results.

[0009] In a fourth aspect, the present application provides a computer-readable storage medium storing a computer program, which is executed by a processor to implement the loop detection method based on active vision and text semantics described in the first aspect.

[0010] In a fifth aspect, the present application provides a computer program product. When the computer program code or instructions are executed on a computer, the computer executes the loop detection method based on text semantics described in the first aspect.

[0011] From the above embodiments of the present application, it can be seen that the present application actively controls the camera through the robotic arm to place it in the optimal viewing angle to accurately collect text features in the environment, and uses it as a marker for loop detection, thereby overcoming the problem that other SLAM methods based on point clouds or images are prone to loop detection errors in environments where environmental features are sparse, degraded or repetitive; at the same time, double loop verification is performed in combination with the point cloud collected by the lidar, which can ensure that the positioning system maintains a high map construction accuracy and robustness of loop detection; finally, the map is optimized based on the loop detection results and graph optimization, further improving the positioning accuracy and robustness of the positioning system. Figure 1 Consistency. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0013] Figure 1 A schematic diagram of the architecture of a positioning system provided in an embodiment of the present application;

[0014] Figure 2 A flowchart of a loop closure detection method based on active vision and text semantics provided in an embodiment of the present application;

[0015] Figure 3 A schematic block diagram of a loop closure detection device based on active vision and text semantics provided in an embodiment of the present application;

[0016] Figure 4 A schematic block diagram of a computer device provided in an embodiment of the present application;

[0017] Figure 5 A schematic block diagram of a computer-readable storage medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] The embodiments of the present application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be understood as limiting the present application. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application without making any creative efforts shall fall within the scope of protection of the present application.

[0019] Figure 1 Figure 1 is a schematic diagram of the architecture of a positioning system 100 based on active vision and text semantics, as proposed in an embodiment of the present application. The active visual SLAM system includes a perception sensor 110, a processing platform 120, a robotic arm 130, and a PID controller 140. The perception sensor 110 is a type of sensor used to sense environmental information around the smart device 100, and the processing platform 120 is used to process images captured by traditional cameras using a classic visual SLAM algorithm to achieve positioning and mapping.

[0020] In the prior art, the perception sensor 110 of the commonly used positioning system is generally a camera, and the classic visual SLAM algorithm specifically includes the following steps: obtaining multiple frames of image data collected by the camera at different times, estimating the position of the camera in the coordinate system of the smart device at different times based on the image data and establishing a map (i.e., the "front end"), and synchronously calculating the similarity between images to obtain loop detection information; optimizing the position and map of the smart device according to the corresponding camera position, map and loop detection information at different times to obtain a globally consistent trajectory and map (i.e., the "back end"); and establishing the map required for the task setting based on the trajectory.

[0021] However, in low-light or dark environments, image methods based solely on traditional cameras cannot extract effective information, and loop detection errors may occur in feature-similar or feature-degraded or dynamic environments, leading to positioning failure. To this end, the present application uses lidar and cameras as perception sensors 110, and provides a loop detection method based on active vision and text semantics provided by one or more embodiments of the present application for use by the processing platform 120 to solve the above problems, so that smart devices can achieve effective positioning in feature-degraded or dynamic environments.

[0022] In one embodiment, the perception sensor 110 includes a laser radar and a camera. When the positioning system 100 is applied to a smart device, the laser radar is mounted on the smart device body to detect the three-dimensional point cloud of the surrounding environment of the smart device 100; the camera is mounted on the smart device body through a robotic arm with six degrees of freedom (DOF), and under the control of a PID controller, the robotic arm can change its posture and drive the camera to collect image data of the required perspective. Among them, the camera can be an infrared camera, a grayscale camera, a color camera or a depth camera, etc. The present application controls the camera to collect images of a specific perspective for semantic extraction and performs loop detection in conjunction with the point cloud collected by the laser radar, which is conducive to achieving feature similarity or feature degradation or accurate positioning in a dynamic environment. It should be noted that the camera is preferably an infrared camera. Due to its passive photosensitivity, it is not affected by factors such as light and is more suitable for use in dark or low-light environments, thereby achieving more accurate positioning.

[0023] The processing platform 120 may include a processor and a memory, wherein the processor is a circuit with signal processing capabilities, and the memory is used to store program instructions corresponding to the loop closure detection method based on active vision and text semantics provided in one or more embodiments of the present application. Some or all of the processors in the processor can call the instructions in the memory to control the laser radar and camera to respectively collect multiple frames of point cloud data and multiple frames of image data and process them according to the loop closure detection method provided in the present application to obtain loop closure detection results.

[0024] In one embodiment, the processor can be a circuit with the ability to read and execute instructions, such as a central processing unit (CPU), a microprocessor, a graphics processing unit (GPU) (which can be understood as a microprocessor), or a digital signal processor (DSP); in another implementation, the processor can implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit is fixed or reconfigurable, for example, the processor is a hardware circuit implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document and implementing the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, the processor can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as a neural network processing unit (NPU), a tensor processing unit (TPU), a deep learning processing unit (DPU), etc.

[0025] In addition, the smart devices in this application may include: road vehicles, water vehicles, air vehicles, industrial equipment, agricultural equipment, wearable devices, mobile robots or entertainment devices, etc. For example, the smart device can be a vehicle, which is a vehicle in a broad sense, and can be a transportation tool (such as a commercial vehicle, a passenger car, a motorcycle, a flying car, a train, etc.), an industrial vehicle (such as a forklift, a trailer, a tractor, etc.), an engineering vehicle (such as an excavator, a bulldozer, a crane, etc.), agricultural equipment (such as a lawn mower, a harvester, etc.), amusement equipment, a toy vehicle, etc. The embodiments of this application do not specifically limit the type of vehicle.

[0026] Figure 2 This is a flow chart of a loop detection method based on active vision and text semantics provided by an embodiment of the present application. The method is applied as follows Figure 1 The positioning system shown is applicable to environments where text elements exist, and specifically includes:

[0027] S210: Control the robotic arm to drive the camera to capture images of the surrounding environment, and detect text information on the captured images. If text information is detected, the PID controller is used to change the position of the end of the robotic arm to move the camera to capture a frontal text image including the text information.

[0028] In one embodiment, the text information includes a string of characters such as words, numbers, and symbols. When detecting text information, the present application changes the posture of the end of the robotic arm through a PID controller to move the camera to capture a front view image including the text information, including: when text is detected in the image, obtaining the text position of the area where the text information is located in the image and the target position where the text should be located in the image (for example, the text target position is the center of the image), calculating the error between the text position and the target position, and using the error and the proportional gain parameter, integral control parameter, and differential gain control parameter of the PID controller to obtain a control signal of the PID controller to change the posture of the end of the robotic arm, thereby adjusting the camera's acquisition angle of view so that the adjusted camera is at an optimal acquisition angle of view, that is, the error between the text position of the area where the text information is located in the captured image and the target position is within a preset error threshold range, so as to capture a front view image including the text information; when the error between the text position and the target position is within the preset error threshold range, controlling the robotic arm to stop moving and synchronously driving the camera to capture a front view text image including the text information, at which time the text position of the area where the text information is located in the front view text image is as close as possible to the target position, to ensure that the camera can capture high-quality text images in a stable state, thereby improving the accuracy and efficiency of subsequent loop detection.

[0029] Specifically, the control signal of the PID controller is calculated using the error and the proportional gain parameter, integral control parameter and differential gain control parameter of the PID controller. Assuming that the error between the text position and the target position is e(t), K_p, K_i and K_d are the proportional, integral and differential control gains respectively, and the control signal of the PID controller is u(t), then

[0030]

[0031] In another embodiment, if no text information is detected in the captured image, the robotic arm is controlled to change the camera's capture angle of view until text information is detected in the captured image. This embodiment uses the robotic arm to control the camera, allowing it to dynamically adjust the camera's capture field of view based on the environment, maximizing the capture of valid text information, thereby improving overall system efficiency and robustness, making this application particularly suitable for use in complex environments such as indoor corridors and underground passages.

[0032] S220: Synchronously controlling the laser radar to collect multiple frames of point clouds, estimating the odometer trajectory using the multiple frames of point clouds, and filtering the multiple frames of three-dimensional point clouds according to the odometer trajectory to obtain at least one frame of key frame point cloud.

[0033] In one embodiment, estimating the odometer trajectory using multiple frames of three-dimensional point clouds includes: extracting features from multiple frames of three-dimensional point clouds to obtain landmark features, such as landmark features such as corner points, plane points, or edge points; using the same landmark features between different frame point clouds to estimate the relative poses between each frame point cloud, including rotation and / or translation, and calculating the cumulative pose of each frame point cloud relative to the starting frame point cloud based on the relative poses between each frame point cloud to obtain the odometer trajectory. It should be noted that the odometer trajectory is composed of the cumulative poses of each frame point cloud relative to the starting frame point cloud and the timestamps corresponding to each frame point cloud. The starting frame point cloud can be the first frame of three-dimensional point cloud collected by the lidar at the initial moment, or it can be the three-dimensional point cloud corresponding to the earlier timestamp in the multiple frame three-dimensional point clouds. In actual application, it can be selected according to the actual situation, and this application does not impose any restrictions on this.

[0034] In one embodiment, filtering multiple frames of three-dimensional point clouds according to the odometer trajectory to obtain key frame point clouds includes: taking the starting frame point cloud as the starting point, obtaining the motion state of the smart device based on the odometer trajectory, and extracting a frame of three-dimensional point cloud as the key frame point cloud every time the motion state of the smart device exceeds a preset distance threshold and / or a preset angle threshold, thereby reducing the loop detection frequency and improving the loop detection efficiency.

[0035] Specifically, the starting frame point cloud is taken as the first key frame point cloud, and the motion state of the smart device is obtained based on the odometer trajectory, that is, the distance change values ​​and / or angle change values ​​corresponding to the remaining frame three-dimensional point clouds and the starting frame point cloud are calculated. If the distance change value and / or angle change value between the point clouds between frames exceeds the preset distance threshold and / or preset angle threshold, the three-dimensional point cloud with a timestamp close to the starting frame point cloud is extracted as the second frame key frame point cloud, and then the second frame key frame point cloud is used as a reference to calculate the distance change values ​​and / or angle change values ​​of the three-dimensional point clouds corresponding to the remaining time stamps. If there is a distance change value and / or angle change value of the point cloud between frames that exceeds the threshold, the three-dimensional point cloud with a timestamp close to the second frame key frame point cloud is extracted as the third frame key frame point cloud, and the remaining time stamp point clouds are extracted by analogy. Key frame point clouds will not be repeated here.

[0036] S230: Select a key text image with the same timestamp as the key frame point cloud from the multiple frames of front-view text images, perform text extraction on the key text image to obtain text information, and convert the current frame point cloud and text information into a global coordinate system.

[0037] Specifically, a global coordinate system is constructed with the base of the robotic arm as the origin. When the camera is used to collect multiple frames of front-facing text images of the surrounding environment, the laser radar simultaneously scans the same surrounding environment of the smart device 100 to obtain multiple frames of three-dimensional point clouds. The laser radar and the camera share the system timestamp of the smart device, and during the operation, the coordinate system of the laser radar and the global coordinate system where the robotic arm base is located are fixed, while the camera coordinate system of the camera will change when the robotic arm moves. For this reason, the point cloud data collected by the laser radar and the images collected by the camera in different camera coordinate systems need to be converted to the same coordinate system for data processing. In this application, the coordinate system of the camera is constructed with the end of the robotic arm being equivalent to the origin of the camera coordinate system.

[0038] In one embodiment, given that the relative positions of the laser radar and the global coordinate system in which the manipulator reside are fixed, a calibration algorithm can be used to obtain relative extrinsic parameters between the laser radar coordinate system and the global coordinate system before use, so that the point cloud data collected by the laser radar can be converted to the global coordinate system using the relative extrinsic parameters. When the camera coordinate system of the camera changes relative to the global coordinate system in which the manipulator base resides, the manipulator, under the control of the processing platform, has a known position of the end of the manipulator relative to the manipulator base. To this end, when the camera is used to collect image data, the camera's extrinsic parameters are obtained, and the image data is converted to the camera coordinate system using the camera's extrinsic parameters to obtain camera coordinates. The camera coordinates are then converted to the global coordinate system using the relative extrinsic parameters between the camera and the manipulator base (i.e., the position of the end of the manipulator relative to the manipulator base) to obtain global coordinates. This ensures that, at the same timestamp, at least a portion of the point cloud data collected by the laser radar has a one-to-one correspondence with a portion of the pixels in the image collected by the camera, i.e., a point in the point cloud data collected by the laser radar can be found in the two-dimensional image collected by the camera, and vice versa.

[0039] In one embodiment, a preset text semantic recognition model is used to extract text from the key text image to obtain text information, and the camera's internal and external parameters are used to convert the text information in the image to a global coordinate system. Assuming that the pixel coordinates of a text in the key text image are (u, v) and its depth is d, the pixel coordinates are converted to the three-dimensional point x_c, y_c, z_c in the camera coordinate system based on the camera's internal parameters, specifically:

[0040]

[0041] Among them, (f x ,f y ) is the focal length of the camera internal parameter, (c x , c y ) is the optical center of the camera internal parameter. Further, through the calibration of the camera and the robot base, the external parameter T cam2base, convert the 3D point in the camera coordinate system into a 3D point in the global coordinate system where the robot base is located:

[0042] P base =T cam2base ·P camera

[0043] Therefore, when the images captured by the camera in different camera coordinate systems are converted to the same coordinate system (i.e., the global coordinate system), it is beneficial to calculate the similarity between the text information in the key text image and the text information obtained at the historical moment.

[0044] It should be noted that the preset text semantic recognition model is a pre-trained neural network model (such as the PaddleOCR model). The text semantic recognition model to be trained can be trained by using images including text and images without text as sample sets, and the training results are constrained by the loss function to obtain the preset text semantic recognition model of this application. This application does not impose any restrictions here.

[0045] S240: Calculate the similarity between the text information in the key text image in the global coordinate system and the historical text information obtained at the historical moment, and determine a candidate loop point cloud based on the similarity. The candidate loop point cloud includes the key frame point cloud and the historical frame point cloud corresponding to the historical text information.

[0046] In one embodiment, the historical text information obtained at the historical moment is a collection of text information obtained based on text detection on images captured at the historical moment, and the historical moment is the moment before the timestamp corresponding to the key text image. Specifically, the similarity between the text information in the key text image and the historical text information obtained at the historical moment in the global coordinate system is calculated, and the candidate loop point cloud is determined based on the similarity, including: performing similarity calculation on the historical text information corresponding to each historical two-dimensional image in the historical two-dimensional image set of the text information in the key text image, if the similarity between the text information in the key text image and the corresponding historical text information in a certain historical image exceeds a preset similarity threshold, it means that the key text image and the historical image form a preliminary loop, and the key frame point cloud corresponding to the key text image and the historical frame point cloud corresponding to the historical image forming the preliminary loop are regarded as candidate loop point clouds; if it is lower than the preset similarity threshold, it means that no loop is formed, and the next frame of the front-facing text image is processed.

[0047] S250: Calculate the geometric consistency of the key frame point cloud and the historical frame point cloud to obtain a distance value, and compare the distance value with a preset distance threshold to obtain a loop detection result.

[0048] Specifically, the Euclidean distance between the keyframe point cloud and the historical frame point cloud obtained at the historical moment is calculated, and the loop detection result is determined based on the Euclidean distance. If the obtained distance value is within the preset distance threshold, the loop detection result indicates that a loop has been formed; otherwise, a loop has not been formed. In addition, extensive experiments have shown that the preset similarity threshold for matching between text information in this application is set to 0.7, and the preset distance threshold for geometric consistency between point clouds is set to 3.0. This ensures that the system can maintain high loop detection accuracy and robustness even when the text information is blurred or noisy.

[0049] After determining the loop closure detection results, the loop closure detection results can also be used to optimize the preliminary map to obtain a global map; wherein the preliminary map is a map obtained by splicing historical frame point clouds collected at historical moments.

[0050] In one embodiment, optimizing a preliminary map in combination with graph optimization and loop closure detection results to obtain a global map includes: using the key frame point cloud corresponding to the earlier timestamp in all key frame point clouds as the starting frame point cloud, calculating the relative transformation relationship between the remaining key frame point clouds and the starting frame point cloud as the pose of each key frame point cloud; constructing a blank pose graph, using the poses of all key frame point clouds as nodes in the pose graph and connecting each node, and the relative pose estimation between adjacent key frames and the detected loop closure detection results provide constraint edges between nodes; using a preset objective function to optimize the position of each node in the pose graph and the constraint edges between each node to obtain the optimal pose of each key frame point cloud, and using the optimal pose to optimize the preliminary map to obtain a global map.

[0051] Specifically, the preset objective function is used to optimize the position of each node in the pose graph and the constraint edges between each node to obtain the optimal pose of each key frame point cloud, including: calculating the loss value of the preset objective function based on the pose of each node in the pose graph and the relative pose between each node. If the loss value is within the preset loss threshold range, it means that the pose of each node in the pose graph is the optimal value, that is, the optimal pose of each key frame point cloud is obtained; conversely, if the loss value is outside the preset loss threshold range, it is necessary to iteratively optimize the pose of each node until the loss value obtained based on the preset objective function is within the preset loss threshold range, so as to reduce the global inconsistency caused by cumulative errors in the positioning system, thereby generating an accurate global map.

[0052] In one embodiment, the preset objective function is:

[0053]

[0054] Among them, ξ n is the pose of the nth node, ΔT ij is the relative transformation of node from pose i to j, Tj T i is the transformation of the node from pose i back to the origin, j T j is the transformation of the node from pose j back to the origin, T is the set of relative poses between all nodes.

[0055] In summary, the application actively controls the camera by the mechanical arm to make it at the optimal viewing angle to accurately collect the text features in the environment, and uses it as a landmark for loop detection, which overcomes the problem that other SLAM methods based on point cloud or image are prone to cause loop detection errors in environments with sparse, degenerate or repetitive features; At the same time, double loop verification is carried out by combining the point cloud collected by the laser radar, which can ensure that the positioning system maintains high map construction accuracy and robustness of loop detection; Finally, the map is optimized based on the loop detection result and graph optimization, which further improves the positioning accuracy and robustness of the positioning system. Figure 1

[0056] Figure 3 is a schematic block diagram of a loop detection device 300 based on active vision and text semantics provided by an embodiment of the application. The device 300 comprises: a first control unit 310, configured to control the mechanical arm to drive the camera to collect images of the surrounding environment, and detect text information from the collected images; if the text information is detected, change the pose of the end of the mechanical arm to move the camera to collect a front-view text image including the text information; a second control unit 320, configured to synchronously control the laser radar to collect multiple frames of point clouds, estimate a odometer trajectory by using the multiple frames of point clouds, and filter the multiple frames of three-dimensional point clouds to obtain at least one frame of key frame point cloud according to the odometer trajectory; a conversion unit 330, configured to select a key text image having the same timestamp as the key frame point cloud from the multiple frames of front-view text images, extract text information from the key text image, and convert the current frame of point cloud and the text information to a global coordinate system; a calculation unit 340, configured to calculate the similarity of the text information of the key text image in the global coordinate system and historical text information obtained at a historical time, determine a candidate loop point cloud according to the similarity, and the candidate loop point cloud comprises a historical frame of point cloud corresponding to the key frame of point cloud and the historical text information; and a detection unit 350, configured to calculate a distance value by performing geometric consistency calculation on the key frame of point cloud and the historical frame of point cloud, and obtain a loop detection result by comparing the distance value with a preset distance threshold.

[0057] For detailed description of the device 300 proposed in the application, please refer to the foregoing embodiments, and the same content will not be repeated here. In some possible embodiments, the processor in the intelligent device 100 described above can include the device 300 described above.

[0058] Figure 4 is a schematic block diagram of a computer device 400 provided by an embodiment of the application. Figure 4 ​The computer device 400 shown includes a memory 410, a processor 420, and a bus 440. Optionally, the computer device 400 also includes a communication interface 430. The memory 410, the processor 420, and the communication interface 430 are communicatively connected to each other via the bus 440.

[0059] The memory 410 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 410 may store programs. When the programs stored in the memory 410 are executed by the processor 420, the processor 420 is configured to perform the various steps of the method according to the embodiments of the present application.

[0060] The processor 420 can be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to implement the method proposed in the embodiment of the present application.

[0061] The processor 420 may also be an integrated circuit chip with signal processing capabilities. During implementation, the various steps of the method described in the embodiments of the present application may be completed by hardware integrated logic circuits in the processor 420 or instructions in software form.

[0062] The above-mentioned processor 420 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The various methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method involved in the embodiments of the present application can be directly reflected as being executed by a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as random access memory, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, registers, etc. The storage medium is located in the memory 410, and the processor 420 reads the information in the memory 410 and completes it in combination with its hardware. Figure 3The units included in the device shown need to perform the functions, or perform the methods described in the method embodiments of the present application.

[0063] The communication interface 430 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the apparatus 400 and other devices or a communication network.

[0064] Bus 440 may include a pathway for transmitting information between various components of device 400 (eg, memory 410 , processor 420 , communication interface 430 ).

[0065] It should be understood that although the above-mentioned device 400 only shows a memory, a processor, and a communication interface, in the specific implementation process, those skilled in the art should understand that the device 400 may also include other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the device 400 may also include hardware devices that implement other additional functions. In addition, those skilled in the art should understand that the device 400 may also include only the devices necessary to implement the embodiments of the present application, and does not necessarily include Figure 4 All devices shown in .

[0066] It should be understood that the processor in the embodiments of the present application may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0067] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0068] Figure 5 It is a schematic block diagram of a computer-readable storage medium 500 provided in an embodiment of the present application. Figure 5 The computer-readable storage medium 500 shown in FIG. 5 stores computer instructions 510. When the computer instructions 510 are executed by a processor, the method corresponding to the above embodiment can be implemented.

[0069] In some possible embodiments, the computer-readable storage medium 500 can be any available medium that can be accessed by a computer, or a data storage device such as a server or data center that includes a collection of one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, or magnetic tapes), optical media (e.g., DVDs), or semiconductor media. The semiconductor media can be a solid-state drive.

[0070] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0071] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0072] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0073] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0074] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0075] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.

[0076] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A loop detection method based on active vision and text semantics, characterized in that: Applied to a positioning system including a PID controller, a camera mounted on a robotic arm, a laser radar, and a processing platform, the method includes: Controlling the robotic arm to drive the camera to capture images of the surrounding environment and detecting text information on the captured images; if text information is detected, changing the position of the end of the robotic arm through the PID controller to move the camera to capture a frontal text image containing text information; Synchronously controlling the laser radar to collect multiple frames of point clouds, estimating an odometer trajectory using the multiple frames of point clouds, and screening the multiple frames of three-dimensional point clouds according to the odometer trajectory to obtain at least one key frame of point cloud; Select a key text image with the same timestamp as the key frame point cloud from the multi-frame front view text image, perform text extraction on the key text image to obtain text information, and convert the current frame point cloud and text information into the global coordinate system; Calculate the similarity between the text information in the key text image in the global coordinate system and the historical text information obtained at the historical moment, and determine the candidate loop point cloud based on the similarity. The candidate loop point cloud includes the key frame point cloud and the historical frame point cloud corresponding to the historical text information; The geometric consistency of the key frame point cloud and the historical frame point cloud is calculated to obtain the distance value, and the distance value is compared with the preset distance threshold to obtain the loop detection result.

2. The loop detection method according to claim 1, wherein: Changing the posture of the end of the robotic arm by the PID controller to move the camera to collect a front view image including text information includes: Obtain the text position of the area where the text information is located in the image and the target position where the text should be located in the image, and calculate the error between the text position and the target position; The error and the proportional gain parameter, integral control parameter, and differential gain control parameter of the PID controller are used to obtain a control signal of the PID controller to change the posture of the end of the robotic arm, thereby adjusting the camera's acquisition angle so that the adjusted camera is at an optimal acquisition angle to capture a front view image including text information.

3. The loop detection method according to claim 1, wherein: The odometry trajectory is estimated using the multi-frame point cloud data, including: Performing feature extraction on the multi-frame point cloud to obtain landmark features; The same landmark features between different frame point clouds are used to estimate the relative poses between each frame point cloud, including rotation and / or translation. The cumulative pose of each frame point cloud relative to the starting frame point cloud is calculated based on the relative poses between the frame point clouds to obtain the odometry trajectory.

4. The loop detection method according to claim 1, wherein: Filtering the multiple frames of three-dimensional point cloud according to the odometer trajectory to obtain at least one key frame point cloud, including: Taking the starting frame point cloud as the starting point, the motion state of the positioning system is obtained based on the odometer trajectory. When the motion state of the positioning system exceeds the preset distance threshold and / or preset angle threshold, a frame of 3D point cloud is extracted from the multi-frame 3D point cloud as the key frame point cloud.

5. The loop detection method according to claim 1, wherein: Convert the current frame point cloud and text information to the global coordinate system, including: Constructing a global coordinate system with the robotic arm base as the origin, obtaining relative external parameters between the laser radar coordinate system and the global coordinate system, and converting the point cloud collected by the laser radar into the global coordinate system; Obtain the intrinsic and extrinsic parameters of the camera, use the intrinsic parameters of the camera to convert the image captured by the camera to the camera coordinate system, and use the relative extrinsic parameters of the camera and the robotic arm base to convert the image captured by the camera from the camera coordinate system to the global coordinate system.

6. The loop detection method according to claim 1, wherein: The calculating the similarity between the text information in the key text image in the global coordinate system and the historical text information obtained at the historical moment, and determining the candidate loop point cloud according to the similarity, includes: The similarity calculation is performed on the historical text information corresponding to each historical two-dimensional image in the text information historical two-dimensional image set in the key text image; wherein: If the similarity between the text information in the key text image and the corresponding historical text information in a certain historical image exceeds a preset similarity threshold, the key frame point cloud corresponding to the key text image and the historical frame point cloud corresponding to the historical image constituting the preliminary loop are regarded as candidate loop point clouds; If the similarity is lower than the preset similarity threshold, it means that no loop is formed, and the next frame of the front-facing text image is processed.

7. The loop detection method according to claim 1, wherein: The geometric consistency of the key frame point cloud and the historical frame point cloud is calculated to obtain the distance value, and the distance value is compared with the preset distance threshold to obtain the loop detection result, including: The Euclidean distance between the key frame point cloud and the historical frame point cloud obtained at the historical moment is calculated, and the loop detection result is determined based on the Euclidean distance. If the obtained distance value is within the preset distance threshold range, the loop detection result indicates that a loop is formed, otherwise it does not form a loop.

8. The loop detection method according to claim 6 or 7, wherein: The preset similarity threshold for matching between text information is set to 0.7, and the preset distance threshold for geometric consistency between point clouds is set to 3.

0.

9. The loop detection method according to claim 1, wherein: After the loopback detection result is determined, the following is also included: The key frame point cloud corresponding to the earlier timestamp among all key frame point clouds is taken as the starting frame point cloud, and the relative transformation relationship between the remaining key frame point clouds and the starting frame point cloud is calculated as the position and posture of each key frame point cloud; Construct a blank pose graph, use the poses of all keyframe point clouds as nodes in the pose graph and connect the nodes. The relative pose estimates between adjacent keyframes and the detected loop detection results provide constraint edges between nodes. The preset objective function is used to optimize the position of each node in the pose graph and the constraint edges between each node to obtain the optimal pose of each keyframe point cloud, and the optimal pose is used to optimize the preliminary map to obtain the global map.

10. A loop detection device based on active vision and text semantics, characterized in that: include: a first control unit, configured to control the robotic arm to drive the camera to capture images of the surrounding environment, and to detect text information in the captured images; if text information is detected, the control unit changes the position of the end of the robotic arm to move the camera to capture a front-view text image containing the text information; A second control unit is used to synchronously control the laser radar to collect multiple frames of point clouds, estimate the odometer trajectory using the multiple frames of point clouds, and filter the multiple frames of three-dimensional point clouds according to the odometer trajectory to obtain at least one key frame point cloud; A conversion unit is used to select a key text image with the same timestamp as the key frame point cloud from multiple frames of front-view text images, perform text extraction on the key text image to obtain text information, and convert the current frame point cloud and text information into a global coordinate system; A calculation unit is used to calculate the similarity between the text information in the key text image and the historical text information obtained at the historical moment in the global coordinate system, and determine the candidate loop point cloud according to the similarity, wherein the candidate loop point cloud includes the key frame point cloud and the historical frame point cloud corresponding to the historical text information; The detection unit is used to calculate the geometric consistency of the key frame point cloud and the historical frame point cloud to obtain a distance value, and compare the distance value with the preset distance threshold to obtain the loop detection result.

11. A positioning system for smart devices, characterized in that: It includes a PID controller, a robotic arm, a camera, a lidar, and a processing platform, including: LiDAR, installed on the smart device itself, is used to detect the point cloud of the smart device's surrounding environment; A camera is mounted on the smart device body via the robotic arm. Under the control of the PID controller, the robotic arm can change its posture to drive the camera to collect image data of a desired viewing angle; A processing platform is used to control the laser radar and the camera to respectively collect the multi-frame point cloud and the multi-frame image and process them according to the loop detection method according to any one of claims 1 to 9 to obtain a loop detection result.

12. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the loop detection method according to any one of claims 1 to 9.

Images