Loopback detection method based on active vision and text semantics and related equipment

Abstract

The invention relates to a loopback detection method based on active vision and text semantics. The loopback detection method comprises the steps that the pose of a mechanical arm is changed to drive a camera to conduct image collection on the surrounding environment to obtain a front view text image; synchronously controlling the laser radar to collect multiple frames of point clouds to screen key frame point clouds; selecting a key text image corresponding to the key frame point cloud from the multiple frames of front view text images, extracting text information in the key text image, and converting the text information into a global coordinate system; calculating the similarity between the text information in the key text image under the global coordinate system and the historical text information obtained at the historical moment, and determining candidate loopback point clouds according to the similarity; and performing geometric consistency calculation on the candidate loopback point clouds to obtain a distance value so as to obtain a loopback detection result. The technical problem of loopback detection errors caused by feature degradation, sparse or similar environments is solved by actively and visually collecting the front view image of the text information and combining the double loopback verification of the point cloud, and the precision and robustness of system positioning and mapping are improved.

Description

Technical Field

[0001] The present application belongs to the field of computer vision technology and relates to a loop detection method and related equipment based on active vision and text semantics. Background Art

[0002] Currently, technologies such as autonomous driving, intelligent robots, metaverse and digital twins are developing rapidly. One of the most important basic technologies among these technologies is Simultaneous Localization and Mapping (SLAM) technology, which aims to enable an intelligent device to determine its position and posture in an unknown environment, thereby further realizing autonomous navigation, control and mapping.

[0003] The key to realizing SLAM technology is the perception and recognition of the environment. Existing technologies usually use LiDAR-based SLAM methods, which mainly rely on the geometric features of point clouds for registration and achieve positioning and map construction by comparing feature points extracted from the environment. However, such methods perform well in static and simple environments, but are often affected by feature loss or mismatch in dynamic or complex environments, resulting in reduced accuracy.

[0004] With the development of science and technology and the increase in demand, SLAM technology has been improved in many aspects, especially through semantic SLAM, active view control (AVC) and loop closure detection to improve positioning accuracy. Among them, semantic SLAM improves the ability to understand the scene by combining semantic information in the environment, such as object categories and attributes, but this type of method relies too much on geometric features, so that when the environmental features are sparse or degraded, the registration accuracy will be significantly reduced, resulting in inaccurate or failed positioning; although active view control can select the optimal perspective during the map construction process to improve the perception of key information, it requires angle-by-angle perspective adjustment to select the optimal angle, which is inefficient; the purpose of loop closure detection is to identify when revisiting the same place to correct the accumulated pose error and ensure the global consistency of the map, but when in a complex or ambiguous scene, loop closure detection is easily affected by mismatching, resulting in map inconsistency, especially in environments with many similar or repeated structures. In view of the above situation, there is an urgent need for a method to solve the problem of positioning and mapping in feature degradation or dynamic environments.

[0005] The contents of the background technology section are merely publicly known technologies and do not necessarily represent the existing technologies in the field. Summary of the invention

[0006] The purpose of the present application is to at least to some extent solve the deficiencies in the prior art. In a first aspect, the present application provides a loop detection method based on active vision and text semantics, comprising: controlling a robotic arm to drive a camera to collect images of the surrounding environment, and performing text information detection on the collected images. If text information is detected, the posture of the end of the robotic arm is changed to move the camera to collect a front view text image including text information; synchronously controlling a laser radar to collect multiple frames of point clouds, using the multiple frames of point clouds to estimate the odometer trajectory, and screening the multiple frames of three-dimensional point clouds according to the odometer trajectory to obtain at least one frame of key frame point clouds; in the multiple frames of front view A key text image with the same timestamp as the key frame point cloud is selected from the text image, text extraction is performed on the key text image to obtain text information, and the current frame point cloud and text information are converted to the global coordinate system; the similarity between the text information in the key text image in the global coordinate system and the historical text information obtained at the historical moment is calculated, and the candidate loop point cloud is determined according to the similarity, and the candidate loop point cloud includes the key frame point cloud and the historical frame point cloud corresponding to the historical text information; the geometric consistency calculation is performed on the key frame point cloud and the historical frame point cloud to obtain the distance value, and the distance value is compared with the preset distance threshold to obtain the loop detection result.

[0007] The second aspect of the present application provides a loop detection device based on active vision and text semantics, the device comprising: a first control unit, used to control a robotic arm to drive a camera to collect images of the surrounding environment, and to detect text information on the collected images. If text information is detected, the posture of the end of the robotic arm is changed to move the camera to collect a front view text image including text information; a second control unit, used to synchronously control a laser radar to collect multiple frames of point clouds, estimate the odometer trajectory using the multiple frames of point clouds, and filter the multiple frames of three-dimensional point clouds according to the odometer trajectory to obtain at least one frame of key frame point cloud; a conversion unit, used to select a key frame from the multiple frames of front view text images; A key text image with the same timestamp as the key frame point cloud is taken, text extraction is performed on the key text image to obtain text information, and the current frame point cloud and text information are converted to a global coordinate system; a calculation unit is used to calculate the similarity between the text information in the key text image in the global coordinate system and the historical text information obtained at the historical moment, and a candidate loop point cloud is determined according to the similarity, the candidate loop point cloud includes the key frame point cloud and the historical frame point cloud corresponding to the historical text information; a detection unit is used to perform geometric consistency calculation on the key frame point cloud and the historical frame point cloud to obtain a distance value, and compare the distance value with a preset distance threshold to obtain a loop detection result.

[0008] The third aspect of the present application provides a positioning system for use on a smart device, including a PID controller robotic arm, a camera, a laser radar and a processing platform, wherein: the laser radar is mounted on the smart device body and is used to sense the three-dimensional point cloud of the surrounding environment of the smart device 100; the camera is installed on the smart device body through a robotic arm, and under the control of the PID controller, the robotic arm can change its own posture and drive the camera to collect image data of the required perspective; the processing platform is used to control the laser radar and the camera to respectively collect multiple frames of point cloud and multiple frames of images and process them according to the loop detection method of the first aspect above to obtain loop detection results.

[0009] In a fourth aspect, the present application provides a computer-readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the loop detection method based on active vision and text semantics described in the first aspect.

[0010] The fifth aspect of the present application provides a computer program product. When the computer program code or instructions are executed on a computer, the computer executes the loop detection method based on text semantics described in the first aspect.

[0011] It can be seen from the above embodiments of the present application that the present application actively controls the camera through the robotic arm to place it in the optimal viewing angle to accurately collect text features in the environment, and uses it as a marker for loop detection, thereby overcoming the problem that other SLAM methods based on point clouds or images are prone to loop detection errors in environments with sparse, degraded or repetitive environmental features; at the same time, double loop verification is performed in combination with the point cloud collected by the lidar to ensure that the positioning system maintains a high level of map construction accuracy and robustness of loop detection; finally, the map is optimized based on the loop detection results and graph optimization, further improving the positioning accuracy and robustness of the positioning system. Figure 1 Consistency. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0013] Figure 1 A schematic diagram of the architecture of a positioning system provided in an embodiment of the present application;

[0014] Figure 2 A flowchart of a loop detection method based on active vision and text semantics provided in an embodiment of the present application;

[0015] Figure 3 A schematic block diagram of a loop detection device based on active vision and text semantics provided in an embodiment of the present application;

[0016] Figure 4 A schematic block diagram of a computer device provided in an embodiment of the present application;

[0017] Figure 5 A schematic block diagram of a computer-readable storage medium provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0019] Figure 1 1 is a schematic diagram of the architecture of a positioning system 100 based on active vision and text semantics proposed in an embodiment of the present application. The active visual SLAM system includes a perception sensor 110, a processing platform 120, a robotic arm 130, and a PID controller 140, wherein the perception sensor 110 is a plurality of sensors for sensing environmental information around the smart device 100, and the processing platform 120 is used to process images collected by a traditional camera through a classic visual SLAM algorithm to achieve positioning and mapping.

[0020] In the prior art, the perception sensor 110 of the commonly used positioning system is generally a camera, and the classic visual SLAM algorithm specifically includes the following steps: obtaining multiple frames of image data collected by the camera at different times, estimating the position and posture of the camera in the coordinate system of the smart device at different times based on the image data and establishing a map (ie, the "front end"), and synchronously calculating the similarity between images to obtain loop detection information; optimizing the position and map of the smart device according to the corresponding camera position, map and loop detection information at different times to obtain a globally consistent trajectory and map (ie, the "back end"); and establishing the map required for the task setting based on the trajectory.

[0021] However, in low-light or dark environments, image methods based only on traditional cameras cannot extract effective information, and loop detection errors may occur in feature-similar or feature-degraded or dynamic environments, leading to positioning failure. To this end, the present application uses a lidar and a camera as a perception sensor 110, and provides a loop detection method based on active vision and text semantics provided by one or more embodiments of the present application for use by the processing platform 120 to solve the above problems, so that smart devices can achieve effective positioning in feature-degraded or dynamic environments.

[0022] In one embodiment, the perception sensor 110 includes a laser radar and a camera. When the positioning system 100 is applied to a smart device, the laser radar is mounted on the smart device body to detect the three-dimensional point cloud of the surrounding environment of the smart device 100; the camera is installed on the smart device body through a mechanical arm with six degrees of freedom (DOF), and under the control of a PID controller, the mechanical arm can change its posture and drive the camera to collect image data of the required viewing angle. Among them, the camera can be an infrared camera, a grayscale camera, a color camera or a depth camera, etc. The present application controls the camera to collect images of a specific viewing angle for semantic extraction and cooperates with the point cloud collected by the laser radar for loop detection, which is conducive to achieving feature similarity or feature degradation or accurate positioning in a dynamic environment. It should be noted that the camera is preferably an infrared camera, because its passive photosensitivity makes it unaffected by factors such as light, and is more suitable for use in dark or low-light environments, thereby achieving more accurate positioning.

[0023] The processing platform 120 may include a processor and a memory, wherein the processor is a circuit with signal processing capabilities, and the memory is used to store program instructions corresponding to the loop closure detection method based on active vision and text semantics provided in one or more embodiments of the present application. Some or all of the processors in the processor can call the instructions in the memory to control the laser radar and camera to respectively collect multiple frames of point cloud data and multiple frames of image data and process them according to the loop closure detection method provided in the present application to obtain loop closure detection results.

[0024] In one embodiment, the processor can be a circuit with the ability to read and run instructions, such as a central processing unit (CPU), a microprocessor, a graphics processing unit (GPU) (which can be understood as a microprocessor), or a digital signal processor (DSP), etc.; in another implementation, the processor can realize certain functions through the logical relationship of the hardware circuit, and the logical relationship of the hardware circuit is fixed or reconfigurable, such as the processor is a hardware circuit implemented by an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, the processor can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as a neural network processing unit (NPU), a tensor processing unit (TPU), a deep learning processing unit (DPU), etc.

[0025] In addition, the smart devices in this application may include: road vehicles, water vehicles, air vehicles, industrial equipment, agricultural equipment, wearable devices, mobile robots or entertainment equipment, etc. For example, the smart device may be a vehicle, which is a vehicle in a broad sense, and may be a vehicle (such as a commercial vehicle, a passenger car, a motorcycle, a flying car, a train, etc.), an industrial vehicle (such as a forklift, a trailer, a tractor, etc.), an engineering vehicle (such as an excavator, a bulldozer, a crane, etc.), agricultural equipment (such as a lawn mower, a harvester, etc.), amusement equipment, a toy vehicle, etc. The embodiment of this application does not specifically limit the type of vehicle.

[0026] Figure 2 is a flow chart of a loop detection method based on active vision and text semantics provided in an embodiment of the present application. The method is applied as follows Figure 1 The positioning system shown is applicable to an environment where text elements exist, and specifically includes:

[0027] S210: Control the robotic arm to drive the camera to capture images of the surrounding environment, and detect text information on the captured images. If text information is detected, the position of the end of the robotic arm is changed through the PID controller to move the camera to capture a frontal text image including the text information.

[0028] In one embodiment, the text information includes a character string such as text, numbers, and symbols. When detecting text information, the present application changes the posture of the end of the robot arm through a PID controller to move the camera to collect a front view image including the text information, including: when text is detected in the image, the text position of the area where the text information is located in the image and the target position where the text should be in the image (for example, the text target position is the center of the image) are obtained, the error between the text position and the target position is calculated, and the control signal of the PID controller is obtained by using the error and the proportional gain parameter, integral control parameter, and differential gain control parameter of the PID controller to change the posture of the end of the robot arm, thereby adjusting the camera's acquisition angle of view, so that the adjusted camera is at an optimal acquisition angle of view, that is, the error between the text position of the area where the text information is located in the image and the target position is within a preset error threshold range, so as to collect a front view image including the text information; when the error between the text position and the target position is within the preset error threshold range, the robot arm is controlled to stop moving and the camera is synchronously driven to collect a front view text image including the text information, at which time the text position of the area where the text information is located in the front view text image is as close to the target position as possible, so as to ensure that the camera can capture high-quality text images in a stable state, thereby improving the accuracy and efficiency of subsequent loop detection.

[0029] Specifically, the control signal of the PID controller is calculated using the error and the proportional gain parameter, integral control parameter and differential gain control parameter of the PID controller. Assuming that the error between the text position and the target position is e(t), K_p, K_i and K_d are the proportional, integral and differential control gains respectively, and the control signal of the PID controller is u(t), then

[0030]

[0031] In another embodiment, if no text information is detected from the captured image, the robotic arm is continuously controlled to change the camera capture angle until text information is detected in the captured image. This embodiment controls the camera through the robotic arm, which can dynamically adjust the camera's capture field of view according to the environment, maximize the acquisition of effective text information, thereby improving the overall system efficiency and robustness, making the application particularly suitable for use in complex indoor corridors, underground passages and other environments.

[0032] S220: synchronously controlling the laser radar to collect multiple frames of point clouds, estimating the odometer trajectory using the multiple frames of point clouds, and filtering the multiple frames of three-dimensional point clouds according to the odometer trajectory to obtain at least one frame of key frame point cloud.

[0033] In one embodiment, estimating the odometer trajectory using multiple frames of three-dimensional point clouds includes: extracting features from multiple frames of three-dimensional point clouds to obtain landmark features, such as landmark features such as corner points, plane points or edge points; using the same landmark features between different frame point clouds to estimate the relative poses between each frame point cloud, including rotation and / or translation, and calculating the cumulative pose of each frame point cloud relative to the starting frame point cloud based on the relative poses between each frame point cloud to obtain the odometer trajectory. It should be noted that the odometer trajectory is composed of the cumulative poses of each frame point cloud relative to the starting frame point cloud and the timestamps corresponding to each frame point cloud. The starting frame point cloud can be the first frame of three-dimensional point cloud collected by the laser radar at the initial moment, or it can be the three-dimensional point cloud corresponding to the earlier timestamp in the multiple frame three-dimensional point clouds. In actual application, it can be selected according to the actual situation, and this application does not limit it here.

[0034] In one embodiment, filtering multiple frames of three-dimensional point clouds according to the odometer trajectory to obtain key frame point clouds includes: taking the starting frame point cloud as the starting point, obtaining the motion state of the smart device based on the odometer trajectory, and extracting a frame of three-dimensional point cloud as the key frame point cloud every time the motion state of the smart device exceeds a preset distance threshold and / or a preset angle threshold, thereby reducing the frequency of loop detection and improving the efficiency of loop detection.

[0035] Specifically, the starting frame point cloud is taken as the first key frame point cloud, and the motion state of the smart device is obtained based on the odometer trajectory, that is, the distance change values ​​and / or angle change values ​​corresponding to the remaining frame three-dimensional point clouds and the starting frame point cloud are calculated. If the distance change value and / or angle change value between the point clouds between frames exceeds the preset distance threshold and / or the preset angle threshold, the three-dimensional point cloud with a timestamp close to the starting frame point cloud is extracted as the second frame key frame point cloud, and then the second frame key frame point cloud is used as a reference to calculate the distance change values ​​and / or angle change values ​​of the three-dimensional point clouds corresponding to the remaining timestamps. If there is a distance change value and / or angle change value of the point cloud between frames that exceeds the threshold, the three-dimensional point cloud with a timestamp close to the second frame key frame point cloud is extracted as the third frame key frame point cloud, and the key frame point clouds of the remaining timestamp point clouds are extracted in the same way, which will not be repeated here.

[0036] S230: Select a key text image having the same timestamp as the key frame point cloud from the multiple frames of front view text images, perform text extraction on the key text image to obtain text information, and convert the current frame point cloud and text information into a global coordinate system.

[0037] Specifically, a global coordinate system is constructed with the base of the robotic arm as the origin. When the camera is used to collect multiple frames of front-facing text images of the surrounding environment, the laser radar simultaneously scans the same surrounding environment of the smart device 100 to obtain multiple frames of three-dimensional point clouds. The laser radar and the camera share the system timestamp of the smart device, and during the operation, the coordinate system of the laser radar and the global coordinate system where the robotic arm base is located are fixed, while the camera coordinate system of the camera will change when the robotic arm moves. For this reason, it is necessary to convert the point cloud data collected by the laser radar and the images collected by the camera in different camera coordinate systems to the same coordinate system for data processing. In this application, the coordinate system of the camera is constructed with the end of the robotic arm being equivalent to the origin of the camera coordinate system.

[0038] In one embodiment, in view of the fact that the relative positions of the laser radar and the global coordinate system where the mechanical arm is located are fixed, the relative external parameters between the laser radar coordinate system and the global coordinate system can be obtained by a calibration algorithm before use, so as to convert the point cloud data collected by the laser radar into the global coordinate system through the relative external parameters. When the camera coordinate system of the camera changes relative to the global coordinate system where the mechanical arm base is located, the movement posture of the end of the mechanical arm with respect to the mechanical arm base is known under the control of the processing platform. Therefore, when the image data is collected by the camera, the internal and external parameters of the camera are obtained, and the image data is converted to the camera coordinate system using the internal parameters of the camera to obtain the camera coordinates, and then the camera coordinates are converted to the global coordinate system through the relative external parameters between the camera and the mechanical arm base (i.e., the movement posture of the end of the mechanical arm with respect to the mechanical arm base) to obtain the global coordinates, so that at the same timestamp, at least part of the point cloud data collected by the laser radar has a one-to-one correspondence with part of the pixels of the image collected by the camera, that is, a point in the point cloud data collected by the laser radar can find a corresponding point on the two-dimensional image collected by the camera, and vice versa.

[0039] In one embodiment, the text information is obtained by extracting the key text image through a preset text semantic recognition model, and the text information in the image is converted to the global coordinate system using the internal and external parameters of the camera. Assuming that the pixel coordinates of a text in the key text image are (u, v) and its depth is d, the pixel coordinates are converted to three-dimensional points x_c, y_c, z_c in the camera coordinate system according to the internal parameters of the camera, specifically:

[0040]

[0041] Among them, (f x ,f y ) is the focal length of the camera internal parameter, (c x , c y ) is the optical center of the camera internal reference. Further, through the calibration of the camera and the robot base, the external reference T cam2base, convert the 3D point in the camera coordinate system into a 3D point in the global coordinate system where the robot base is located:

[0042] P base =T cam2base ·P camera

[0043] Therefore, when the images captured by the camera in different camera coordinate systems are converted to the same coordinate system (ie, the global coordinate system), it is beneficial to calculate the similarity between the text information in the key text image and the text information obtained at the historical moment.

[0044] It should be noted that the preset text semantic recognition model is a pre-trained neural network model (such as the PaddleOCR model), which can be trained by using images including text and images without text as sample sets to train the text semantic recognition model to be trained, and constraining the training results through a loss function to obtain the preset text semantic recognition model of this application. This application is not limited here.

[0045] S240: Calculate the similarity between the text information in the key text image in the global coordinate system and the historical text information obtained at the historical moment, and determine the candidate loop point cloud according to the similarity, wherein the candidate loop point cloud includes the key frame point cloud and the historical frame point cloud corresponding to the historical text information.

[0046] In one embodiment, the historical text information obtained at the historical moment is a set of text information obtained based on text detection of the images collected at the historical moment, and the historical moment is the moment before the timestamp corresponding to the key text image. Specifically, the similarity between the text information in the key text image and the historical text information obtained at the historical moment in the global coordinate system is calculated, and the candidate loop point cloud is determined according to the similarity, including: calculating the similarity of the historical text information corresponding to each historical two-dimensional image in the historical two-dimensional image set of the text information in the key text image, if the similarity between the text information in the key text image and the corresponding historical text information in a certain historical image exceeds a preset similarity threshold, it means that the key text image and the historical image form a preliminary loop, and the key frame point cloud corresponding to the key text image and the historical frame point cloud corresponding to the historical image forming the preliminary loop are regarded as candidate loop point clouds; if it is lower than the preset similarity threshold, it means that no loop is formed, and the next frame of the front view text image is processed.

[0047] S250: Calculate the geometric consistency of the key frame point cloud and the historical frame point cloud to obtain a distance value, and compare the distance value with a preset distance threshold to obtain a loop detection result.

[0048] Specifically, the Euclidean distance between the key frame point cloud and the historical frame point cloud obtained at the historical moment is calculated and the loop detection result is determined based on the Euclidean distance; wherein, if the distance value obtained is within the preset distance threshold range, the loop detection result indicates that a loop is formed, otherwise it does not form a loop. In addition, a large number of experiments have shown that the preset similarity threshold for matching between text information in this application is set to 0.7, and the preset distance threshold for geometric consistency between point clouds is set to 3.0, which can ensure that the system can maintain a high loop detection accuracy and robustness when the text information is blurred or noisy.

[0049] After determining the loop detection result, the loop detection result can also be used to optimize the preliminary map to obtain a global map; wherein the preliminary map is a map obtained by splicing historical frame point clouds collected at historical moments.

[0050] In one embodiment, optimizing a preliminary map in combination with graph optimization and loop detection results to obtain a global map includes: taking the key frame point cloud corresponding to an earlier timestamp among all key frame point clouds as the starting frame point cloud, calculating the relative transformation relationship between the remaining key frame point clouds and the starting frame point cloud as the pose of each key frame point cloud; constructing a blank pose graph, taking the poses of all key frame point clouds as nodes in the pose graph and connecting the nodes, and providing constraint edges between nodes based on the relative pose estimation between adjacent key frames and the detected loop detection results; optimizing the positions of each node in the pose graph and the constraint edges between each node using a preset objective function to obtain the optimal pose of each key frame point cloud, and using the optimal pose to optimize the preliminary map to obtain a global map.

[0051] Specifically, the preset objective function is used to optimize the position of each node in the pose graph and the constraint edges between each node to obtain the optimal pose of each key frame point cloud, including: calculating the loss value of the preset objective function based on the pose of each node in the pose graph and the relative pose between each node. If the loss value is within the preset loss threshold range, it means that the pose of each node in the pose graph is the optimal value, that is, the optimal pose of each key frame point cloud is obtained; conversely, if the loss value is outside the preset loss threshold range, it is necessary to iteratively optimize the pose of each node until the loss value obtained based on the preset objective function is within the preset loss threshold range, so as to reduce the global inconsistency caused by cumulative errors in the positioning system, thereby generating an accurate global map.

[0052] In one embodiment, the preset objective function is:

[0053]

[0054] Among them, ξ n is the position of the nth node, ΔT ij is the relative transformation of node from position i to j, Tj is the transformation of the node from position i back to the origin, T j is the transformation of the node from pose j back to the origin, is the relative pose set between all nodes.

[0055] In summary, the present application uses a robotic arm to actively control the camera to place it at the optimal viewing angle to accurately collect text features in the environment, and uses it as a marker for loop detection, thereby overcoming the problem that other SLAM methods based on point clouds or images are prone to loop detection errors in environments with sparse, degraded or repetitive environmental features; at the same time, double loop verification is performed in combination with the point cloud collected by the lidar to ensure that the positioning system maintains a high level of map construction accuracy and robustness of loop detection; finally, the map is optimized based on the loop detection results and graph optimization, further improving the positioning accuracy and robustness of the positioning system Figure 1 Consistency.

[0056] Figure 3 It is a schematic block diagram of a loop detection device 300 based on active vision and text semantics provided in an embodiment of the present application. The device 300 includes: a first control unit 310, which is used to control the robot arm to drive the camera to collect images of the surrounding environment, and detect text information on the collected images. If text information is detected, the posture of the end of the robot arm is changed to move the camera to collect a front view text image including text information; a second control unit 320, which is used to synchronously control the laser radar to collect multiple frames of point clouds, use the multiple frames of point clouds to estimate the odometer trajectory and filter the multiple frames of three-dimensional point clouds according to the odometer trajectory to obtain at least one frame of key frame point cloud; a conversion unit 330, which is used to select a point cloud with the same time as the key frame point cloud in the multiple frames of front view text images. A key text image with an interval stamp is used to extract text from the key text image to obtain text information and convert the current frame point cloud and the text information into a global coordinate system; a calculation unit 340 is used to calculate the similarity between the text information in the key text image in the global coordinate system and the historical text information obtained at the historical moment, and determine a candidate loop point cloud according to the similarity, the candidate loop point cloud includes the key frame point cloud and the historical frame point cloud corresponding to the historical text information; a detection unit 350 is used to perform geometric consistency calculation on the key frame point cloud and the historical frame point cloud to obtain a distance value, and compare the distance value with a preset distance threshold to obtain a loop detection result.

[0057] For detailed description of the apparatus 300 proposed in the present application, please refer to the above embodiments, and the same contents will not be repeated here. In some possible embodiments, the processor in the above smart device 100 may include the above apparatus 300.

[0058] Figure 4 It is a schematic block diagram of a computer device 400 provided in an embodiment of the present application. Figure 4The computer device 400 shown includes a memory 410, a processor 420 and a bus 440. Optionally, the computer device 400 also includes a communication interface 430. The memory 410, the processor 420 and the communication interface 430 are connected to each other through the bus 440.

[0059] The memory 410 may be a read only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 410 may store a program. When the program stored in the memory 410 is executed by the processor 420, the processor 420 is used to execute each step of the method proposed in the embodiment of the present application.

[0060] The processor 420 can adopt a general-purpose central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits to execute relevant programs to implement the method proposed in the embodiment of the present application.

[0061] The processor 420 may also be an integrated circuit chip having signal processing capabilities. During implementation, each step of the method described in the embodiments of the present application may be completed by an integrated logic circuit of hardware in the processor 420 or instructions in software form.

[0062] The processor 420 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The steps of the methods involved in the embodiments of the present application may be directly embodied as being executed by a hardware decoding processor, or may be executed by a combination of hardware and software modules in the decoding processor. The software module may be located in a mature storage medium in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, or the like. The storage medium is located in the memory 410, and the processor 420 reads the information in the memory 410 and completes the processing in combination with its hardware. Figure 3The units included in the device shown need to perform the functions, or perform the methods described in the method embodiments of the present application.

[0063] The communication interface 430 uses a transceiver device such as, but not limited to, a transceiver to implement communication between the apparatus 400 and other devices or a communication network.

[0064] Bus 440 may include a pathway for transmitting information between various components of device 400 (eg, memory 410 , processor 420 , communication interface 430 ).

[0065] It should be understood that although the above device 400 only shows a memory, a processor, and a communication interface, in the specific implementation process, those skilled in the art should understand that the device 400 may also include other devices necessary for normal operation. At the same time, according to specific needs, those skilled in the art should understand that the device 400 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the device 400 may also include only the devices necessary to implement the embodiments of the present application, and does not necessarily include Figure 4 All devices shown in .

[0066] It should be understood that the processor in the embodiment of the present application may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0067] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0068] Figure 5 It is a schematic block diagram of a computer-readable storage medium 500 provided in an embodiment of the present application. Figure 5 The computer-readable storage medium 500 shown in the figure stores computer instructions 510. When the computer instructions 510 are executed by a processor, the method corresponding to the above embodiment can be implemented.

[0069] In some possible embodiments, the computer-readable storage medium 500 may be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more available media sets. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium may be a solid state drive.

[0070] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0071] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0072] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0073] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0074] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0075] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application.

[0076] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A loop detection method based on active vision and text semantics, characterized in that: Applied to a positioning system including a PID controller, a camera mounted on a robotic arm, a laser radar, and a processing platform, the method includes: Control the mechanical arm to drive the camera to collect images of the surrounding environment, and detect text information on the collected images. If text information is detected, the PID controller is used to change the posture of the end of the mechanical arm to move the camera to collect a front view text image including text information; Synchronously controlling the laser radar to collect multiple frames of point clouds, estimating odometer trajectories using the multiple frames of point clouds, and screening the multiple frames of three-dimensional point clouds according to the odometer trajectories to obtain at least one frame of key frame point clouds; A key text image having the same timestamp as the key frame point cloud is selected from multiple frames of front-view text images, text is extracted from the key text image to obtain text information, and the current frame point cloud and text information are converted to the global coordinate system; Calculate the similarity between the text information in the key text image in the global coordinate system and the historical text information obtained at the historical moment, and determine the candidate loop point cloud according to the similarity, wherein the candidate loop point cloud includes the key frame point cloud and the historical frame point cloud corresponding to the historical text information; The geometric consistency of the key frame point cloud and the historical frame point cloud is calculated to obtain the distance value, and the distance value is compared with the preset distance threshold to obtain the loop detection result.

2. The loop detection method according to claim 1, characterized in that: Changing the posture of the end of the mechanical arm by the PID controller to move the camera to collect a front view image including text information includes: Obtain the text position of the area where the text information is located in the image and the target position where the text should be located in the image, and calculate the error between the text position and the target position; The control signal of the PID controller is obtained by utilizing the error and the proportional gain parameter, integral control parameter and differential gain control parameter of the PID controller to change the posture of the end of the robotic arm, thereby adjusting the acquisition angle of the camera so that the adjusted camera is at the optimal acquisition angle to acquire a front view image including text information.

3. The loop detection method according to claim 1, characterized in that: The odometer trajectory is estimated using the multi-frame point cloud data, including: Performing feature extraction on the multi-frame point cloud to obtain landmark features; The same landmark features between different frame point clouds are used to estimate the relative poses including rotation and / or translation between each frame point cloud, and the cumulative pose of each frame point cloud relative to the starting frame point cloud is calculated according to the relative poses between each frame point cloud to obtain the odometer trajectory.

4. The loop detection method according to claim 1, characterized in that: The multiple frames of three-dimensional point clouds are screened according to the odometer trajectory to obtain at least one frame of key frame point cloud, including: Taking the starting frame point cloud as the starting point, the motion state of the positioning system is obtained based on the odometer trajectory. When the motion state of the positioning system exceeds a preset distance threshold and / or a preset angle threshold, a frame of three-dimensional point cloud is extracted from multiple frames of three-dimensional point cloud as a key frame point cloud.

5. The loop detection method according to claim 1, characterized in that: Convert the current frame point cloud and text information to the global coordinate system, including: A global coordinate system is constructed with the robotic arm base as the origin, relative external parameters between the laser radar coordinate system and the global coordinate system are obtained, and the point cloud collected by the laser radar is converted to the global coordinate system; The internal and external parameters of the camera are obtained, and the image captured by the camera is converted into a camera coordinate system using the internal parameters of the camera, and the image captured by the camera is converted from the camera coordinate system to a global coordinate system using the relative external parameters of the camera and the robot base.

6. The loop detection method according to claim 1, characterized in that: The calculating the similarity between the text information in the key text image in the global coordinate system and the historical text information obtained at the historical moment, and determining the candidate loop point cloud according to the similarity, includes: The similarity of the historical text information corresponding to each historical two-dimensional image in the text information historical two-dimensional image set in the key text image is calculated; wherein: If the similarity between the text information in the key text image and the corresponding historical text information in a certain historical image exceeds a preset similarity threshold, the key frame point cloud corresponding to the key text image and the historical frame point cloud corresponding to the historical image constituting the preliminary loop are regarded as candidate loop point clouds; If the similarity is lower than the preset similarity threshold, it means that no loop is formed, and the next frame of the front view text image is processed.

7. The loop detection method according to claim 1, characterized in that: The geometric consistency of the key frame point cloud and the historical frame point cloud is calculated to obtain the distance value, and the distance value is compared with the preset distance threshold to obtain the loop detection result, including: The Euclidean distance between the key frame point cloud and the historical frame point cloud obtained at the historical moment is calculated, and the loop detection result is determined based on the Euclidean distance; wherein, if the obtained distance value is within the preset distance threshold range, the loop detection result indicates that a loop is formed, otherwise it does not form a loop.

8. The loop detection method according to claim 6 or 7, characterized in that: The preset similarity threshold for matching between text information is set to 0.7, and the preset distance threshold for geometric consistency between point clouds is set to 3.

0.

9. The loop detection method according to claim 1, characterized in that: After the loop detection result is determined, it also includes: The key frame point cloud corresponding to the earlier timestamp among all key frame point clouds is taken as the starting frame point cloud, and the relative transformation relationship between the remaining key frame point clouds and the starting frame point cloud is calculated as the position and pose of each key frame point cloud; Construct a blank pose graph, use the poses of all keyframe point clouds as nodes in the pose graph and connect the nodes. The relative pose estimation between adjacent keyframes and the detected loop detection results provide constraint edges between nodes. The preset objective function is used to optimize the position of each node in the pose graph and the constraint edges between the nodes to obtain the optimal pose of each keyframe point cloud, and the optimal pose is used to optimize the preliminary map to obtain the global map.

10. A loop detection device based on active vision and text semantics, characterized in that: include: A first control unit is used to control the mechanical arm to drive the camera to collect images of the surrounding environment, and to detect text information on the collected images. If text information is detected, the posture of the end of the mechanical arm is changed to move the camera to collect a front view text image including text information; A second control unit is used to synchronously control the laser radar to collect multiple frames of point clouds, estimate the odometer trajectory using the multiple frames of point clouds, and filter the multiple frames of three-dimensional point clouds according to the odometer trajectory to obtain at least one frame of key frame point cloud; A conversion unit, used to select a key text image with the same timestamp as the key frame point cloud from multiple frames of front-view text images, perform text extraction on the key text image to obtain text information, and convert the current frame point cloud and text information into a global coordinate system; A calculation unit is used to calculate the similarity between the text information in the key text image in the global coordinate system and the historical text information obtained at the historical moment, and determine the candidate loop point cloud according to the similarity. The candidate loop point cloud includes the key frame point cloud and the historical frame point cloud corresponding to the historical text information. The detection unit is used to calculate the geometric consistency of the key frame point cloud and the historical frame point cloud to obtain a distance value, and compare the distance value with a preset distance threshold to obtain a loop detection result.

11. A positioning system for smart devices, characterized in that: It includes PID controller, robotic arm, camera, lidar and processing platform, including: A laser radar, mounted on the smart device body, for detecting a point cloud of the surrounding environment of the smart device 100; The camera is installed on the smart device body through the mechanical arm. Under the control of the PID controller, the mechanical arm can change its posture to drive the camera to collect image data of the required viewing angle; A processing platform is used to control the laser radar and the camera to respectively collect the multi-frame point cloud and the multi-frame image and process them according to the loop detection method according to any one of claims 1 to 9 to obtain a loop detection result.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is executed by a processor to implement the loop detection method according to any one of claims 1 to 9.

Images