Loop detection method and device, loop detection robot, and storage medium

Abstract

The embodiments of the present application disclose a loop detection method and device, a loop detection robot, and a storage medium. The method includes: acquiring an image frame sequence, each image frame in the image frame sequence includes laser point cloud data and temperature array data; performing feature extraction on each image frame in the image frame sequence to obtain image features corresponding to each image frame; marking image frames whose number of image features exceeds a preset threshold as key frames, and determining corresponding posture information based on the laser point cloud data and temperature array data included in the key frames; performing loop detection based on the first matching result between the key frame and the historical frame, and the second matching result between the posture information and the historical posture information. The method introduces temperature array data in the process of loop detection, thereby improving the success rate of loop detection.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and specifically to a loop detection method and device, a loop detection robot, and a storage medium. Background Art

[0002] China has many data centers, and the energy consumption of refrigeration equipment is hundreds of millions every year. However, traditional data center energy-saving measures have poor flexibility and low precision, so it is particularly important to build a data center energy consumption optimization model based on big data and machine learning. The basis of model training is big data, but the digital conditions of existing computer rooms are uneven. Temperature and humidity measurement points often require multiple cabinets or even multiple cabinets to share one, and air conditioning control parameters are often not obtained in time, and even need to contact on-site front-line personnel to take photos for confirmation. These have laid obstacles for the promotion of intelligent IDC (Internet Data Center). Therefore, there is an urgent need for a method with strong scalability that can avoid the cumbersome engineering of installing sensors to collect data. Inspection robots came into being, but the computer room scenes are highly similar and have few features, which is not conducive to robot positioning and mapping. It is easy to misjudge the robot's own position and easily cause false loops.

[0003] In summary, an improved and optimized loop detection method is urgently needed to help inspection robots locate and map the computer room scene. Summary of the invention

[0004] To solve the above technical problems, the embodiments of the present application respectively provide a loop detection method and device, a loop detection robot, and a computer-readable storage medium, and introduce temperature dot matrix data to optimize loop detection, thereby improving the success rate of loop detection.

[0005] According to one aspect of an embodiment of the present application, a loop detection method is provided, the method comprising:

[0006] Acquire an image frame sequence, wherein each image frame in the image frame sequence includes laser point cloud data and temperature array data; perform feature extraction on each image frame in the image frame sequence to obtain image features corresponding to each image frame; mark image frames whose number of image features exceeds a preset threshold as key frames, and determine corresponding posture information based on the laser point cloud data and temperature array data included in the key frames; perform loop detection based on a first matching result between the key frame and a historical frame, and a second matching result between the posture information and the historical posture information.

[0007] In another exemplary embodiment, the method further includes: storing image frames whose number of image features does not exceed a preset threshold as historical frames.

[0008] In another exemplary embodiment, before performing loop detection based on the first matching result between the key frame and the historical frame, and the second matching result between the posture information and the historical posture information, the method further includes: inputting the key frame into the autoencoder network for encoding to obtain the encoding value of the key frame; inputting the key frame into the autoencoder network for encoding to obtain the encoding value of the key frame; matching the encoding value of the key frame with the encoding value of the historical frame to obtain the first matching result.

[0009] In another exemplary embodiment, the step of inputting the key frame into an autoencoding network for encoding to obtain the encoded value of the key frame includes: inputting the key frame into an autoencoding network for residual processing and convolution processing to obtain the encoded value of the key frame.

[0010] In another exemplary embodiment, the loop detection is performed based on a first matching result between the key frame and the historical frame, and a second matching result between the posture information and the historical posture information, including: if the first matching result and the second matching result both indicate a successful match, determining that a loop is established.

[0011] In another exemplary embodiment, the loop detection is performed based on a first matching result between the key frame and the historical frame, and a second matching result between the posture information and the historical posture information, including: if the first matching result indicates a matching failure, determining that the loop fails, and entering the encoding value of the key frame into a database for storing the encoding values ​​of historical frames.

[0012] In another exemplary embodiment, the method further includes: if the second matching result indicates a matching failure, entering the posture information of the key frame into a database for storing historical posture information.

[0013] According to one aspect of an embodiment of the present application, a loop detection device is provided, comprising: an acquisition module, configured to acquire an image frame sequence, each image frame in the image frame sequence comprising laser point cloud data and temperature array data; an extraction module, configured to perform feature extraction on each image frame in the image frame sequence to obtain image features corresponding to each image frame; a marking module, configured to mark image frames whose number of image features exceeds a preset threshold as key frames, and determine corresponding posture information based on the laser point cloud data and temperature array data included in the key frames; and a matching module, configured to perform loop detection based on a first matching result between the key frame and a historical frame, and a second matching result between the posture information and the historical posture information.

[0014] In another embodiment, the loop detection device further includes a storage module configured to store image frames whose number of image features does not exceed a preset threshold as historical frames.

[0015] In another embodiment, the loop detection device further includes an encoding module configured to input the key frame into the self-encoding network for encoding to obtain an encoding value of the key frame.

[0016] The matching module is configured to match the encoding value of the key frame with the encoding value of the historical frame to obtain a first matching result, and perform loop detection based on the first matching result between the key frame and the historical frame, and the second matching result between the posture information and the historical posture information.

[0017] In another embodiment, the encoding module further includes an auto-encoding processing unit configured to input the key frame into the auto-encoding network for residual processing and convolution processing to obtain the encoding value of the key frame.

[0018] In another embodiment, the matching module includes a matching success unit configured to determine that a loop is established if both the first matching result and the second matching result indicate a successful match.

[0019] In another embodiment, the loop detection device further includes a storage module configured to determine loop failure if the first matching result indicates a matching failure, and enter the encoding value of the key frame into a database for storing encoding values ​​of historical frames.

[0020] In another embodiment, based on the storage module, a configuration can be added on the basis of its original configuration: if the second matching result indicates a match failure, the posture information of the key frame is entered into a database for storing historical posture information. According to one aspect of an embodiment of the present application, a loop detection device is provided, the device includes a mobile robot, characterized in that the robot includes: at least 4 temperature sensors, the sensors are evenly distributed at the same height position of the device, for collecting temperature data; a laser measuring instrument, for collecting laser point cloud data; an odometer, for collecting posture information; a controller, electrically connected to the temperature sensor, the laser measuring instrument and the odometer, and reads pre-stored computer-readable instructions to execute the method as described above.

[0021] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor of a computer, the computer executes the method as described above.

[0022] According to one aspect of the embodiments of the present application, a computer program product or a computer program is also provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above method.

[0023] In the technical solution provided in the embodiments of the present application, loop detection is performed based on laser point cloud data. Taking into account that there are certain differences in ambient temperatures corresponding to different postures, temperature array data is introduced in the process of loop detection. The temperature differences in different postures can assist in optimizing the loop detection method. Therefore, the corresponding posture information is determined based on the laser point cloud data and the temperature array data, which is conducive to positioning and mapping, avoiding misjudgment of the position during the detection process, thereby improving the success rate of loop detection.

[0024] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0026] Figure 1 is a schematic diagram of an exemplary loop detection process;

[0027] Figure 2 is a schematic diagram of another exemplary loop closure detection process;

[0028] Figure 3 is a flow chart of a loop detection method shown in an exemplary embodiment of the present application;

[0029] Figure 4 is a schematic diagram of the distribution of an exemplary robot temperature sensor;

[0030] Figure 5 is a flowchart of a self-encoding processing method shown in another exemplary embodiment of the present application;

[0031] Figure 6 is a schematic diagram of the structure of an auto-encoding network shown in an exemplary embodiment of the application;

[0032] Figure 7 is a loop detection flow chart showing an exemplary application scenario of the present application;

[0033] Figure 8 is a structural schematic diagram of a loop detection device shown in an exemplary embodiment of the present application;

[0034] Fig. 9 An exemplary embodiment of the present application shows a schematic diagram of the structure of a computer system of a loop detection device. DETAILED DESCRIPTION

[0035] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the attached claims.

[0036] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0037] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0038] The term "multiple" as used in this application refers to two or more than two. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.

[0039] First, the real machine room scene is single, and there are many long corridor scenes, which is one of the situations that the inspection robot SLAM (Simultaneous localization and mapping) cannot effectively handle, and it is easy to have the problem of false loops in laser SLAM. The technical solution of this application introduces temperature information combined with posture-assisted loop detection to avoid the occurrence of false loops. Secondly, the temperature data information obtained by a single temperature probe is less and there is a slight fluctuation. The robustness of using temperature as a feature is poor, so the concept of temperature array is proposed to introduce spatial structure information for floating-point temperature information to enhance robustness.

[0040] Loop detection: Establishing a pose constraint relationship between the current frame and a previous frame is called loop closure; if the two frames are close enough, the pose of the next frame can be recursively inferred from the previous frame, and a constraint relationship between the two frames can definitely be established. It should be noted that two frames that are far apart may not necessarily establish such a constraint relationship. The process of finding historical frames that can establish such a pose constraint relationship is loop detection, which is explained below with an example:

[0041] like Figure 1 As shown, Figure 1 This is an exemplary diagram of the loop detection process. Among them, the pose constraints are all established with the previous frame, and the pose error of the fifth frame has accumulated the errors of the previous four constraints. But at this time, if we find that the pose of the fifth frame does not have to be inferred from the fourth frame, it can also be inferred from the second frame, because there are only two constraint errors, and the calculation error is obviously smaller. Establishing a pose constraint relationship with a previous frame like this is called a loop. Loop closure reduces the cumulative error by reducing the number of constraints.

[0042] However, the pose of the fifth frame is not necessarily inferred from the second frame, such as Figure 2 As shown, Figure 2 This is another example of the loop detection process. The first and third frames may be used to calculate the pose of the second frame. The reason why we use the previous frame to recursively infer the pose of the next frame is that the two frames are close enough to establish a constraint between the two frames, but the two frames that are far away may not be able to establish such a constraint relationship. Finding the historical frames that can establish such a pose constraint is loop detection.

[0043] See also Figure 3 , Figure 3 : is a flowchart of a loop detection method shown in an exemplary embodiment of the present application. The execution subject of the method may be a controller, and the controller controls the loop detection device to perform loop detection according to the method. The controller is located in a movable loop detection device. During the movement of the movable loop detection device, the controller may control the acquisition device in the mobile loop detection device to acquire an image frame sequence. The loop detection method includes at least steps S310 to S340, which are described in detail below:

[0044] S310: Acquire an image frame sequence, where each image frame in the image frame sequence includes laser point cloud data and temperature array data.

[0045] An image frame refers to the smallest unit that constitutes a video. In this embodiment, the image frame sequence is composed of continuous image frames, which can be roughly understood as constituting a video, wherein each image frame includes laser point cloud data and temperature array data.

[0046] Laser point cloud refers to the information of position, distance, etc. that is reflected when a laser beam is irradiated onto the surface of an object. If the laser beam is scanned along a certain trajectory, the reflected laser point information will be recorded while scanning. Since the scanning is extremely precise, a large number of laser points can be obtained, thus densely forming a cloud of laser points.

[0047] Temperature array refers to the temperature data of points with a certain spatial order and structure in this application. For example, at the same height, the temperature data obtained from points in different directions, the points in different directions constitute a spatial plane, which constitutes the temperature array data. Figure 4 , Figure 4 This is a schematic diagram of the distribution of an exemplary robot temperature sensor, in which there are 12 temperature measurement points, from temperature measurement point A to temperature measurement point L, evenly distributed on the surface of the robot, and every four temperature measurement points are at the same height, forming a temperature measurement point plane, for example, temperature measurement point A, temperature measurement point B, temperature measurement point C, and temperature measurement point D are all at the same height, forming a temperature measurement point plane, and each of the above measurement points is installed with a temperature sensor for collecting temperature values. On the other hand, the points for obtaining temperature data can also form a three-dimensional image, for example, a cube formed by 8 points for obtaining temperature data, for example, temperature measurement points A to temperature measurement point H, form a cube, and the temperature array of the present application can be formed according to the temperature data taken from its 8 endpoints.

[0048] S320: Extract features from each image frame in the image frame sequence to obtain image features corresponding to each image frame.

[0049] Feature extraction is a concept in computer vision and image processing. It refers to using computers to extract image information and determine whether each image point belongs to an image feature. The result of feature extraction is to divide the points on the image into different subsets. These subsets often belong to isolated points, continuous curves or continuous areas. Subsequent steps will use these subsets to determine whether they meet the preset standards. In the feature extraction process, it can be determined whether it meets the image feature based on predefined marks or symbols. For example, when the number of straight lines in the image frame is greater than 3, it has an image feature, or when the number of straight lines in the image frame is greater than 3 and the number of curves is greater than 2, it is defined as having an image feature. That is, the preset conditions of image features are not necessarily single conditions. Some need to meet at least two preset conditions to be defined as image features.

[0050] S330: Marking an image frame whose number of image features exceeds a preset threshold as a key frame, and determining corresponding posture information according to the laser point cloud data and temperature array data included in the key frame.

[0051] This embodiment screens the image frames according to a preset threshold value. If the number of image features of a certain image frame exceeds the preset threshold value, the image frame is marked as a key frame that meets the definition of this embodiment. It is worth noting that only the image frames marked as key frames need to determine the corresponding posture information based on the laser point cloud data and temperature array data included therein, and different laser point cloud data and temperature array data correspond to different posture information. For example, if the preset threshold value is 9, and the number of image features of the image frame obtained in step S320 is 10, then the image frame is marked as a key frame. Only key frames can be matched with historical frames, and the corresponding posture information determined by the laser point cloud data and temperature array data included in the key frame is matched with the historical posture. Marking the image frame whose number of image features exceeds the preset threshold as a key frame can be understood as a pre-screening of the pre-set data, so that the method can avoid a large number of useless matches in the subsequent matching process, reducing the workload while improving the accuracy of loop detection. Assuming that there are 30,000 image frames in total, and all image frames are directly matched with historical frames, and their posture information is matched with historical frames, at least 30,000 matching operations will be performed, which increases the processing of marking key frames. If there are 5,000 key frames in total, subsequent matching will be performed at most 5,000 times, which greatly reduces the number of subsequent matching operations and the workload of data comparison. In addition, the key frames have undergone image feature extraction, and frames with high feature correlation can be used as the basis for subsequent matching, thereby improving the accuracy of loop detection.

[0052] S340: Perform loop detection based on the first matching result between the key frame and the historical frame, and the second matching result between the posture information and the historical posture information.

[0053] If an image frame is marked as a key frame, it is matched with the historical frame to obtain the frame matching result, i.e., the first matching result, and the corresponding posture information determined by the laser point cloud data and the temperature array data is matched with the historical posture information to obtain the posture matching result, i.e., the second matching result, and then loop detection is performed based on the first matching result and the second matching result. For example, the historical frame identical to the key frame A is searched in the database used to store historical frames. If it exists, the first matching result indicates success. If it does not exist, the first matching result indicates failure. The match can be verified according to the digital code. Then, the posture information A corresponding to the key frame A is obtained, and the database stored in the historical posture information is searched to see whether there is historical posture information identical to the posture information A. If so, the second matching result indicates success; if not, the second matching result indicates failure. The process can be matched based on symbolic marks, for example, a circle is used to represent the posture information A. If there is a circle in the database of historical posture information, the posture information of the key frame successfully matches the historical posture information, and the second matching result indicates a successful match. Here, only an example is given to illustrate how to obtain the first and second matching results, and it is not limited to using digital codes or symbolic marks as the basis for the matching process.

[0054] This embodiment performs loop detection based on laser point cloud data. Considering that there are certain differences in ambient temperatures corresponding to different postures, temperature array data is introduced in the process of loop detection. Because there are certain differences in ambient temperatures corresponding to different postures, the temperature differences of different postures can assist in optimizing the loop detection method. Therefore, the corresponding posture information is determined based on the laser point cloud data and the temperature array data, which is conducive to positioning and mapping, avoiding misjudgment of the position during the detection process, and thus improving the success rate of loop detection. .

[0055] In another embodiment, the loop detection method further includes: storing image frames whose number of image features does not exceed a preset threshold as historical frames.

[0056] For example, if the number of features of a certain image frame is 3 and the preset threshold is 5, the number of features of the image frame does not exceed the preset threshold, and it does not meet the key frame defined in this embodiment. The image frame is stored in the historical frame database as a historical frame. In some embodiments, for subsequent rapid comparison, the historical frame will be encoded to obtain the encoding value of the historical frame, which can reduce the amount of data comparison and make subsequent loop detection faster.

[0057] This embodiment illustrates that image frames whose number of image features does not exceed a preset threshold are stored in a database for storing historical frames, thereby updating the historical frame database and adding the historical frames as a reference for the next matching, thereby making the loop detection method faster and more accurate.

[0058] See also Figure 5 , Figure 5 is a flowchart of a self-encoding processing method shown in another exemplary embodiment of the present application. Before step S340, the loop detection method further includes at least steps S510 to S520, which are described in detail as follows:

[0059] S510: Input the key frame into the self-encoding network for encoding to obtain the encoding value of the key frame.

[0060] This step encodes the key frame in binary, which also involves the code value database. For example, key frame A is input into the autoencoder network for encoding, and the binary code obtained is 1010, then 1010 is stored in the code value database, and key frame A uniquely corresponds to the code 1010 in the code value database.

[0061] S520: Match the encoding value of the key frame with the encoding value of the historical frame to obtain a first matching result.

[0062] For example, when the coding value 1010 of the key frame A mentioned above is matched with the coding value of the historical frame, there will be two situations: successful matching or failed matching. A successful matching indicates that the key frame is the same as the historical frame, that is, the database storing the coding values ​​of the historical frames is searched for the coding value 1010. If the coding value 1010 exists in the database, the first matching result indicates a successful matching. If the coding value 1010 does not exist in the database, the first matching result indicates a failed matching.

[0063] This embodiment performs encoding processing on the image frame, that is, matching is performed according to the encoding value, so that subsequent data processing is simpler and faster. The encoding network can reduce the cost of manually extracting features and has good reusability.

[0064] In another embodiment, step S310 at least further includes: inputting the key frame into the autoencoder network for residual processing and convolution processing to obtain the encoding value of the key frame. The depth of the deep learning network has a great influence on the final classification and recognition effect, so the general idea is to design the network as deep as possible, but in fact it is not the case. The stacking of conventional networks (plain network) is getting worse and worse when the network is very deep. One of the reasons is that the deeper the network, the more obvious the gradient disappears, and the training effect of the network will not be very good, that is, as the network performance decreases with the increase of depth. However, the shallow network (shallower network) cannot significantly improve the recognition effect of the network, so the problem to be solved now is how to solve the problem of gradient disappearance while deepening the network. This embodiment introduces a residual module to solve this problem, so that after the key frame is processed by convolution, the performance of the autoencoder network will not decrease with the increase of depth, ensuring the stability of the autoencoder network.

[0065] See also Figure 6 , Figure 6 It is a structural diagram of an autoencoder network shown in an exemplary embodiment of the present application, which includes a two-dimensional convolution and residual module, that is, the key frame is input into the autoencoder network for residual processing and convolution processing, and finally filtered to obtain the binary code value of the key frame.

[0066] This embodiment further illustrates how key frames are processed in the autoencoder network, and introduces two-dimensional convolution and residual modules. The role of the two-dimensional convolution is to weighted sum the neighborhood of each pixel of the key frame (the neighborhood size is the size of the kernel) to obtain the output value of the pixel point. After subsequent processing by the residual module, the encoding value of the key frame can be better filtered.

[0067] In another embodiment, step S340 includes the following content: if both the first matching result and the second matching result indicate a successful match, it is determined that the loop is established.

[0068] Only when both the first matching result and the second matching result are successful, the loop is established, that is, the key frame matches the historical frame, and the corresponding posture information determined by the laser point cloud data and temperature array data included in the key frame matches the historical posture information.

[0069] In another embodiment, step S340 includes the following contents: if the first matching result indicates a matching failure, determining that the loop fails, and entering the encoding value of the key frame into a database for storing encoding values ​​of historical frames.

[0070] If the coding value of the key frame does not match the coding value of the historical frame, the first matching result is mismatch. The coding value of the key frame is entered into the database of the coding value of the historical frame and used as the coding value of the historical frame. This can reduce the pre-step of data comparison and make the loop detection method faster.

[0071] In another embodiment, based on S340, the loop detection method further includes: if the second matching result indicates a matching failure, entering the posture information of the key frame into a database for storing historical posture information.

[0072] If the laser point cloud data and temperature array data included in the key frame determine that the corresponding posture information does not match the historical posture information, it means that it is a scene that has never been visited. This posture information is entered into the database of historical posture information. If it matches later, it means that the scene has been reached.

[0073] This embodiment provides a method for updating a database of historical posture information in real time, entering posture information that does not match historical posture information into the database, increasing the amount of historical posture information, and increasing the reference amount for matching of loop detection, making the detection more accurate.

[0074] In another embodiment, reference Figure 7 , Figure 7 is a loop detection flow chart showing an exemplary application scenario of the present application. The exemplary loop detection process includes at least steps S710 to S770, which are described in detail as follows:

[0075] S710: Acquire an image frame sequence, where each image frame in the image frame sequence includes laser point cloud data and temperature array data.

[0076] The controller controls the acquisition device to obtain an image frame sequence. The acquisition device here refers to a device that can continuously obtain each image frame, and it is necessary to ensure that each image frame includes laser point cloud data and temperature array data. For detailed process, please refer to the description of the above step S310.

[0077] S720: Perform feature extraction on each image frame in the image frame sequence to obtain image features corresponding to each image frame.

[0078] By performing feature extraction on a single image frame, corresponding image features and a corresponding number of features can be obtained, and subsequent judgments can be made based on the number of features. For the detailed process, please refer to the description of S320 above.

[0079] S730: Mark the image frame whose number of image features exceeds a preset threshold as a key frame, and determine the corresponding posture information according to the laser point cloud data and temperature array data included in the key frame.

[0080] If the number of image features obtained in the above step S720 exceeds the preset threshold, the current image frame is marked as a key frame, and the key frame laser point cloud data and temperature array data are obtained to determine the corresponding posture information, and the posture information is used for subsequent judgment matching. Marking the image frame whose number of image features exceeds the preset threshold as a key frame can be understood as a pre-screening of the front data, so that the method can avoid a large number of useless matches in the subsequent matching process, reduce the workload and improve the accuracy of loop detection. For detailed process, refer to the description of the above S330.

[0081] S740: Input the key frame into the self-encoding network for encoding to obtain the encoding value of the key frame.

[0082] After the key frame is processed by the self-encoding network, the corresponding binary code value can be obtained, that is, the code value corresponds to the key frame one by one, and the code value can be used for direct matching later, which is more convenient and quick. For the detailed process, please refer to the description of S510 above.

[0083] S750: Determine whether the encoding value of the key frame matches the encoding value of the historical frame.

[0084] This step is similar to the above S520, matching the key frame code value with the historical frame code value to obtain the first matching result. For example, if the key frame code value is 1010, then find whether there is a historical code value of 1010 in the database storing the code values ​​of the historical frames; if there is 1010, it means the match is successful, if not, it means the match fails.

[0085] S760a: If the encoding value is judged to be a match, determine whether the posture information of the key frame matches the historical posture information.

[0086] If the first matching result indicates a successful match, then it is determined whether the key frame's pose information matches the historical pose information to obtain a second matching result. For example, the key frame's code value is 1110, and the code 1110 exists in the historical frame's code value database, which means that the key frame's code value matches the historical frame's code value successfully, and then the key frame's pose information is matched with the historical pose information in the historical pose information database to obtain a second matching result.

[0087] S760b: If the code value is judged not to match, the loop fails, and the code value of the key frame is entered into the database of the code value of the historical frame.

[0088] If the first matching result indicates that the match fails, the loop fails, and the code value of the key frame is entered into the database for storing the code value of the historical frame. For example, the code value of the key frame is 1101, and the matching result indicates that the match fails, indicating that there is no code value 1101 in the database for storing the code value of the historical frame, then the loop fails, and the code value 1101 is entered into the database for storing the code value of the historical frame. At this time, the code value 1101 is stored in the database.

[0089] S770a: If the posture information is judged to be a match, the loop is successful. If the second matching result indicates a successful match, the loop is successful. This is the case where both the first matching result and the second matching result indicate a success. The loop can only be established when both matches are successful. For example, the encoding value 1101 of the key frame matches the encoding value of the historical frame in the database. If the encoding value of a historical frame in the database is also 1101, the first matching result indicates success; then the key frame and the historical frame with the encoding value of 1101 are matched. If the posture information of the key frame with the encoding value of 1101 matches the posture information of the historical frame with the encoding value of 1101, the second matching result indicates a successful match.

[0090] S770b: If the posture information is judged to be mismatched, the loop fails, and the posture information of the key frame is entered into a database for storing historical posture information.

[0091] If the second matching result indicates a match failure, the loop fails, and the pose information of the key frame is entered into the database for storing historical pose information. For example, if the first matching result indicates a successful match, that is, the key frame's code value 1101 can match the code value in the database of the historical frame's code value, but its pose information cannot match the pose information of the historical frame, that is, the second matching result indicates a match failure, the loop fails, and the pose information of the key frame with a code value of 1101 is entered into the database for storing historical pose information.

[0092] This embodiment systematically illustrates that there is a certain order in which the first matching result and the second matching result are obtained, and there is also an order in which the two judgments are made. Finally, if the loop fails, it means that a scene that the loop detection device or device has not been to has appeared. If the loop succeeds, it means that the scene has not been visited by the loop detection device or device, thereby improving the accuracy of the loop detection.

[0093] The present application also provides a loop detection device, such as Figure 8 As shown, Figure 8 : is a schematic diagram of the structure of a loop detection device shown in an exemplary embodiment of the present application. The loop detection device includes:

[0094] An acquisition module 810 is configured to acquire an image frame sequence, each image frame in the image frame sequence includes laser point cloud data and temperature array data;

[0095] An extraction module 820 is configured to perform feature extraction on each image frame in the image frame sequence to obtain image features corresponding to each image frame;

[0096] The marking module 830 is configured to mark the image frame whose number of image features exceeds a preset threshold as a key frame, and determine the corresponding posture information according to the laser point cloud data and temperature array data included in the key frame;

[0097] The matching module 840 is configured to perform loop detection based on a first matching result between the key frame and the historical frame, and a second matching result between the posture information and the historical posture information.

[0098] In another embodiment, the loop detection device further includes a storage module configured to store image frames whose number of image features does not exceed a preset threshold as historical frames.

[0099] In another embodiment, the loop detection device further includes an encoding module configured to input the key frame into the self-encoding network for encoding to obtain an encoding value of the key frame.

[0100] The matching module 840 is configured to match the encoding value of the key frame with the encoding value of the historical frame to obtain a first matching result, and perform loop detection based on the first matching result between the key frame and the historical frame, and the second matching result between the posture information and the historical posture information.

[0101] In another embodiment, the encoding module further includes an auto-encoding processing unit configured to input the key frame into the auto-encoding network for residual processing and convolution processing to obtain the encoding value of the key frame.

[0102] In another embodiment, the matching module 840 includes a matching success unit configured to determine that a loop is established if both the first matching result and the second matching result indicate a successful match.

[0103] In another embodiment, the loop detection device further includes a storage module configured to determine loop failure if the first matching result indicates a matching failure, and enter the encoding value of the key frame into a database for storing encoding values ​​of historical frames.

[0104] In another embodiment, based on the above storage module, a configuration can be further added on the basis of its original configuration: if the second matching result indicates a matching failure, the posture information of the key frame is entered into a database for storing historical posture information.

[0105] It should be noted that the loop detection device provided in the above embodiment and the loop detection method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment and will not be repeated here.

[0106] The embodiment of the present application further provides a loop detection device, the device comprising a mobile robot, comprising: at least 4 temperature sensors, the sensors being evenly distributed at the same height position of the device, for collecting temperature data;

[0107] Laser measuring instrument, used to collect laser point cloud data;

[0108] Odometer, used to collect posture information;

[0109] The controller is electrically connected to the temperature sensor, the laser measuring instrument and the odometer, and executes the loop detection method in each of the above embodiments by reading pre-stored computer-readable instructions.

[0110] like Figure 4 As shown, 12 temperature sensors are installed at 12 temperature measurement points, and the minimum requirement is 4, that is, temperature sensors are only installed at the same height around the robot. For example, temperature sensors are installed at points A, B, C and D respectively.

[0111] This embodiment selects the temperature feature to determine whether the robot is looping, thereby avoiding the situation where a pure laser method is prone to false loops due to the single topological structure of the computer room, and can help the mobile robot assist in mapping in laser-degraded conditions such as long corridors.

[0112] The loop detection equipment can also be used in the data collection process of IDC (Internet Data Center) energy-saving projects, which can save labor costs while ensuring data accuracy.

[0113] See also Fig. 9 , Fig. 9 1 is a schematic diagram of the structure of a computer system of a loop detection device shown in an exemplary embodiment of the present application, which shows a schematic diagram of the structure of a computer system of a loop detection device suitable for implementing an embodiment of the present application.

[0114] It should be noted that Fig. 9 The computer system 900 of the loop detection device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0115] like Fig. 9 As shown, the computer system 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 902 or the program loaded from the storage part 908 to the random access memory (RAM) 903, such as executing the method in the above embodiment. In the RAM 903, various programs and data required for system operation are also stored. The CPU 901, ROM 902 and RAM 903 are connected to each other through a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0116] The following components are connected to the I / O interface 905: an input section 906 including a keyboard, a mouse, etc.; an output section 907 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 910 as needed so that a computer program read therefrom is installed into the storage section 908 as needed.

[0117] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 909, and / or installed from a removable medium 911. When the computer program is executed by a central processing unit (CPU) 901, various functions defined in the system of the present application are executed.

[0118] It should be noted that the computer-readable medium shown in the embodiment of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it. In the present application, a computer-readable signal medium can include a data signal propagated in a baseband or as a part of a carrier wave, wherein a computer-readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0119] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0120] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0121] Another aspect of the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements the loop detection method as described above when the computer program is executed by a processor. The computer-readable storage medium may be included in the loop detection device described in the above embodiment, or may exist independently without being assembled into the loop detection device.

[0122] Another aspect of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the loop detection method provided in each of the above embodiments.

[0123] According to one aspect of an embodiment of the present application, a computer system is also provided, including a central processing unit (CPU), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) or a program loaded from a storage portion into a random access memory (RAM), such as executing the method in the above embodiment. In RAM, various programs and data required for system operation are also stored. CPU, ROM and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0124] The following components are connected to the I / O interface: an input part including a keyboard, a mouse, etc.; an output part including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker; a storage part including a hard disk, etc.; and a communication part including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication part performs communication processing via a network such as the Internet. A drive is also connected to the I / O interface as needed. Removable media, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., are installed on the drive as needed so that the computer program read therefrom is installed into the storage part as needed.

[0125] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. A person skilled in the art can easily make corresponding changes or modifications based on the main concept and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.

Claims

1. A loop detection method, characterized in that: The method comprises: Acquire an image frame sequence, wherein each image frame in the image frame sequence includes laser point cloud data and temperature array data; wherein the temperature array data refers to temperature data of points having a certain spatial order and structure; Extracting features from each image frame in the image frame sequence to obtain image features corresponding to each image frame; Marking an image frame whose number of image features exceeds a preset threshold as a key frame, and determining corresponding posture information according to the laser point cloud data and temperature array data included in the key frame; Inputting the key frame into an autoencoding network for encoding to obtain an encoding value of the key frame; The encoding value of the key frame is matched with the encoding value of the historical frame to obtain a first matching result, and loop detection is performed based on the first matching result between the key frame and the historical frame and the second matching result between the posture information and the historical posture information.

2. The method according to claim 1, characterized in that The method further comprises: The image frames whose number of image features does not exceed a preset threshold are stored as historical frames.

3. The method according to claim 1, characterized in that The step of inputting the key frame into the autoencoding network for encoding to obtain the encoding value of the key frame includes: The key frame is input into the autoencoder network for residual processing and convolution processing to obtain the encoding value of the key frame.

4. The method according to claim 1, characterized in that The loop detection is performed according to the first matching result between the key frame and the historical frame, and the second matching result between the posture information and the historical posture information, including: If both the first matching result and the second matching result indicate a successful match, it is determined that a loop is established.

5. The method according to claim 1, characterized in that The loop detection is performed according to the first matching result between the key frame and the historical frame, and the second matching result between the posture information and the historical posture information, including: If the first matching result indicates a matching failure, loop failure is determined, and the encoding value of the key frame is entered into a database for storing encoding values ​​of historical frames.

6. The method according to claim 5, characterized in that The method further comprises: If the second matching result indicates that the matching fails, the posture information of the key frame is entered into a database for storing historical posture information.

7. A loop detection device, characterized in that: include: An acquisition module is configured to acquire an image frame sequence, each image frame in the image frame sequence includes laser point cloud data and temperature array data; wherein the temperature array data refers to temperature data of points with a certain spatial order and structure; An extraction module, configured to perform feature extraction on each image frame in the image frame sequence to obtain image features corresponding to each image frame; A marking module, configured to mark an image frame whose number of image features exceeds a preset threshold as a key frame, and determine corresponding posture information according to the laser point cloud data and temperature array data included in the key frame; An encoding module, configured to input the key frame into an autoencoding network for encoding to obtain an encoding value of the key frame; The matching module is configured to match the encoding value of the key frame with the encoding value of the historical frame to obtain a first matching result, and perform loop detection based on the first matching result between the key frame and the historical frame, and the second matching result between the posture information and the historical posture information.

8. A loop detection device, comprising a mobile robot, characterized in that: The robot comprises: At least 4 temperature sensors, which are evenly distributed at the same height of the device to collect temperature data; Laser measuring instrument, used to collect laser point cloud data; Odometer, used to collect posture information; A controller is electrically connected to the temperature sensor, the laser measuring instrument and the odometer, and executes the method according to any one of claims 1 to 6 by reading pre-stored computer-readable instructions.

9. A computer-readable storage medium, characterized in that: Computer-readable instructions are stored thereon, and when the computer-readable instructions are executed by a processor of a computer, the computer is caused to execute the method according to any one of claims 1 to 6.

Images