Target image frame selection method and device suitable for robot exploration scene

Abstract

The invention provides a target image frame selection method and device suitable for a robot exploration scene. The target image frame selection method suitable for the robot exploration scene comprises the steps of obtaining a local feature image frame of image data according to a pre-generated convolutional neural network and the image data collected by a robot; generating a global descriptor of the image data according to a pre-generated shrinkage auto-encoder and the local feature image frame; and selecting a target image frame of the image data according to the global descriptor and a visual angle change threshold range of the robot. According to the invention, a new key frame selection method is designed to ensure long-term and short-term data association so as to overcome adverse effects on robot exploration operation in a complex exploration scene, thereby effectively improving the efficiency and precision of positioning and map construction of the robot in the exploration operation.

Description

Technical Field

[0001] The present application belongs to the field of oil and gas exploration technology, in particular to the field of seismic exploration data processing technology, and specifically relates to a target image frame selection method and device suitable for robot exploration scenarios. Background Art

[0002] With the continuous development of robot technology, robots are increasingly used in oil and gas reservoir exploration and detection tasks. In these tasks, robots need to complete tasks by sensing the environment, building maps, planning paths, and other steps. Among them, visual positioning and mapping technology are very important. For both, completing the observation and collection of surrounding environment information is the prerequisite for robots to be used in exploration operations. The exploration operation environment of robots is often more complex. High-speed, irregularly moving objects and drastic changes in light intensity that often appear in application scenarios often affect the performance of robot exploration operations. Summary of the invention

[0003] The present invention belongs to the field of computer vision. One purpose of the present invention is to provide a target image frame selection method suitable for robot exploration scenarios. The method is based on a visual inertial SLAM (Simultaneous Localization and Mapping) robot system and provides a new key frame selection method. The method can efficiently and robustly enhance data association, help the robot system to achieve robust positioning and global consistent map building functions, thereby further improving the positioning accuracy of the robot and enhancing the robot's applicability in the field of exploration operations.

[0004] Another object of the present invention is to provide a target image frame selection device suitable for robot exploration scenarios. Another object of the present invention is to provide an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the steps of the target image frame selection method suitable for robot exploration scenarios when executing the computer program. Another object of the present invention is to provide a readable medium, on which a computer program is stored, and when the computer program is executed by the processor, the steps of the target image frame selection method suitable for robot exploration scenarios are implemented.

[0005] In order to solve the technical problems in the background technology of this application, the present invention provides the following technical solutions:

[0006] In a first aspect, the present invention provides a target image frame selection method suitable for a robot exploration scenario, comprising:

[0007] Acquire a local feature image frame of the image data according to the image data collected by the robot and a pre-generated convolutional neural network;

[0008] Generate a global descriptor of the image data based on the pre-generated contraction autoencoder and the local feature image frame;

[0009] A target image frame of the image data is selected according to the global descriptor and a threshold range of a viewing angle change of the robot.

[0010] In one embodiment of the present invention, the convolutional neural network model is a VGG-16 model, and obtaining a local feature image frame of the image data according to the image data collected by the robot and a pre-generated convolutional neural network includes:

[0011] Extracting a descriptor of a current image frame of the image data according to the VGG-16 model;

[0012] The local feature image frame is acquired according to the descriptor and the VGG-16 model.

[0013] In one embodiment of the present invention, generating a global descriptor of the image data according to the pre-generated contraction autoencoder and the local feature image frame includes:

[0014] Generating a gradient histogram of the local feature image frame according to the contraction autoencoder;

[0015] The partial feature image frame is reconstructed according to the contraction autoencoder and the gradient histogram to generate the global descriptor.

[0016] In one embodiment of the present invention, the target image frame selection method applicable to the robot exploration scene also includes:

[0017] The global descriptor is normalized.

[0018] In one embodiment of the present invention, the step of generating the convolutional neural network includes:

[0019] Generate training data and verification data based on the history of the image data, and perform the following iterative operations:

[0020] Training a pre-constructed initial model or training model of the convolutional neural network according to the training data, wherein the training model is generated by training the initial model according to the training data;

[0021] Verifying the training model according to the verification data to generate a verification result;

[0022] When the verification result reaches a preset training cutoff condition, the training is stopped to generate the convolutional neural network;

[0023] The contraction autoencoder consists of 3 sub-encoders and 3 sub-decoders.

[0024] In one embodiment of the present invention, selecting a target image frame of the image data according to the global descriptor and a threshold range of a viewing angle change of the robot includes:

[0025] generating disparity between adjacent image frames according to the global descriptor;

[0026] Selecting a candidate image frame of the target image frame according to the disparity and the viewing angle change threshold range;

[0027] In the candidate image frame, determining a candidate score of the current image frame according to the number of inliers of the current image frame and an image frame previous to the current image frame;

[0028] The target image frame is selected from the candidate image frames according to the candidate scores.

[0029] In one embodiment of the present invention, the target image frame selection method applicable to the robot exploration scene also includes:

[0030] Determining an acceleration threshold of the robot according to geomorphic data of the exploration block where the robot is located and acquisition requirement data;

[0031] When the acceleration of the robot exceeds the acceleration threshold, the target image frame is reselected.

[0032] In a second aspect, the present invention provides a target image frame selection device suitable for a robot exploration scene, the device comprising:

[0033] A local feature image frame acquisition module, used to acquire a local feature image frame of the image data based on the image data collected by the robot and a pre-generated convolutional neural network;

[0034] A global descriptor generation module, used to generate a global descriptor of the image data according to the pre-generated contraction autoencoder and the local feature image frame;

[0035] The target image frame selection module is used to select the target image frame of the image data according to the global descriptor and the viewing angle change threshold range of the robot.

[0036] In one embodiment of the present invention, the convolutional neural network model is a VGG-16 model, and the local feature image frame acquisition module includes:

[0037] A descriptor extraction unit, configured to extract a descriptor of a current image frame of the image data according to the VGG-16 model;

[0038] A local feature image frame acquisition unit is used to acquire the local feature image frame according to the descriptor and the VGG-16 model.

[0039] In one embodiment of the present invention, the global descriptor generation module includes:

[0040] A gradient histogram generating unit, configured to generate a gradient histogram of the local feature image frame according to the contraction autoencoder;

[0041] A global descriptor generating unit is used to reconstruct the local feature image frame according to the contraction autoencoder and the gradient histogram to generate the global descriptor.

[0042] In one embodiment of the present invention, a target image frame selection device suitable for a robot exploration scene also includes:

[0043] The global descriptor normalization module is used to perform normalization processing on the global descriptor.

[0044] In one embodiment of the present invention, the target image frame selection device applicable to the robot exploration scene further includes: a convolutional neural network generation module, which is used to generate the convolutional neural network, and the convolutional neural network generation module includes:

[0045] The training data generating unit is used to generate training data and verification data according to the history of the image data, and perform the following iterative operations:

[0046] A model training unit, configured to train a pre-constructed initial model or a training model of the convolutional neural network according to the training data, wherein the training model is generated by training the initial model according to the training data;

[0047] A verification result generating unit, used to verify the training model according to the verification data to generate a verification result;

[0048] A convolutional neural network generation unit, configured to stop training when the verification result reaches a preset training cutoff condition to generate the convolutional neural network;

[0049] The contraction autoencoder consists of 3 sub-encoders and 3 sub-decoders.

[0050] In one embodiment of the present invention, the target image frame selection module includes:

[0051] a disparity generating unit, configured to generate disparity between adjacent image frames according to the global descriptor;

[0052] A candidate image frame selection unit, configured to select a candidate image frame of a target image frame according to the disparity and the viewing angle change threshold range;

[0053] A candidate score determination unit, configured to determine, within the candidate image frame, a candidate score of the current image frame according to the number of inliers of the current image frame and an image frame preceding the current image frame;

[0054] A target image frame selection unit is used to select the target image frame from the candidate image frames according to the candidate scores.

[0055] In one embodiment of the present invention, a target image frame selection device suitable for a robot exploration scene also includes:

[0056] An acceleration threshold setting module, used to determine the acceleration threshold of the robot according to the topographic data of the exploration block where the robot is located and the acquisition demand data;

[0057] The target image frame reselection module is used to reselect the target image frame when the acceleration of the robot exceeds the acceleration threshold.

[0058] In a third aspect, the present invention provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of a target image frame selection method suitable for a robot exploration scenario.

[0059] In a fourth aspect, the present invention provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of a target image frame selection method applicable to a robot exploration scenario are implemented.

[0060] In a fifth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of a target image frame selection method applicable to a robot exploration scenario.

[0061] From the above description, it can be seen that an embodiment of the present invention provides a target image frame selection method and device suitable for a robot exploration scenario. The corresponding target image frame selection method suitable for a robot exploration scenario includes: first, obtaining a local feature image frame of image data based on a pre-generated convolutional neural network and image data collected by the robot; then, generating a global descriptor of the image data based on the pre-generated contraction autoencoder and the local feature image frame; finally, selecting a target image frame of the image data based on the global descriptor and the robot's perspective change threshold range.

[0062] The corresponding target image frame selection device suitable for robot exploration scenarios includes: a local feature image frame acquisition module, which is used to acquire local feature image frames of image data based on a pre-generated convolutional neural network and image data collected by the robot; a global descriptor generation module, which is used to generate a global descriptor of image data based on a pre-generated contraction autoencoder and local feature image frames; a target image frame selection module, which is used to select target image frames of image data based on the global descriptor and the robot's viewing angle change threshold range.

[0063] The target image frame selection method and device suitable for robot exploration scenarios provided in the embodiments of the present invention can better improve the performance of robots in the fields of exploration, development and operations, and provide a key frame selection method and device for robots applied to visual inertial SLAM, which can effectively improve the performance of multi-sensor robots in the field of exploration operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0065] Figure 1 A schematic diagram of a flow chart of a target image frame selection method applicable to a robot exploration scenario in an embodiment of the present invention;

[0066] Figure 2 1 is a flow chart of step 100 of a method for selecting a target image frame in a robot exploration scenario in an embodiment of the present invention;

[0067] Figure 3 It is a flowchart diagram of step 200 of a method for selecting a target image frame in a robot exploration scenario in an embodiment of the present invention;

[0068] Figure 4 Another flowchart of a method for selecting a target image frame in a robot exploration scenario according to an embodiment of the present invention;

[0069] Figure 5 A third flow chart of a method for selecting a target image frame in a robot exploration scenario according to an embodiment of the present invention;

[0070] Figure 6 It is a flowchart diagram of step 500 of a method for selecting a target image frame in a robot exploration scenario in an embodiment of the present invention;

[0071] Figure 7 It is a flowchart diagram of step 300 of a method for selecting a target image frame in a robot exploration scenario in an embodiment of the present invention;

[0072] Figure 8 It is a fourth flow chart of a method for selecting a target image frame in a robot exploration scenario in an embodiment of the present invention;

[0073] Fig. 9 It is a flowchart of a method for selecting a target image frame in a robot exploration scenario in a specific embodiment of the present invention;

[0074] Fig.10 A mind map of a target image frame selection method applicable to a robot exploration scenario in a specific embodiment of the present invention;

[0075] Fig.11 The flowchart of step S3 in the specific implementation mode of the present invention is as follows:

[0076] Fig.12 It is a block diagram of a target image frame selection device suitable for a robot exploration scene in a specific embodiment of the present invention;

[0077] Fig.13 It is a schematic diagram of the structure of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0078] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0079] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0080] It should be noted that the terms "including" and "having" in the specification and claims of the present application and the above-mentioned drawings and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. In the absence of conflict, the embodiments in the present application and the features in the embodiments may be combined with each other. The present application will be described in detail below with reference to the drawings and in combination with the embodiments.

[0081] The acquisition, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of laws and regulations.

[0082] Embodiment 1:

[0083] In the field of computer vision, the key frame selection methods used in SLAM systems mainly include the following:

[0084] (1) Fixed interval method: select key frames at a fixed frame interval. This method is simple and easy to implement, but it often selects too many key frames, resulting in increased computational complexity and reduced accuracy of map construction and pose estimation.

[0085] (2) Motion threshold method: The key frame is selected by setting a certain motion threshold. When the robot's movement distance exceeds the threshold, the current frame is used as the key frame. This method can effectively reduce the amount of calculation, but it may ignore some special cases, such as the robot being stationary for a long time.

[0086] (3) Disparity threshold method: The key frame is selected by setting a certain disparity threshold. When the disparity between the current frame and the previous key frame exceeds the threshold, the current frame is used as the new key frame. This method can ensure that there is a large difference in perspective between the selected key frames, but the effect may not be ideal when the scene changes slightly or there is a large amount of overlap between adjacent frames.

[0087] (4) Local map information method: The key frames are selected by analyzing the local map information (such as SURF features). This method can select key frames adaptively, but it needs to calculate the local map information, which increases the amount of calculation.

[0088] In addition to the above commonly used key frame selection strategies, there are also some machine learning-based methods that automatically select key frames with the assistance of neural networks. For example, convolutional neural networks are used to extract features from images and use classifiers to determine whether they are key frames; or reinforcement learning and other methods are used to continuously optimize key frame selection strategies through interaction with the environment. However, these methods are still in the research stage and have great development potential.

[0089] Based on the above reasons, an embodiment of the present invention provides a specific implementation of a target image frame selection method suitable for a robot exploration scenario, see Figure 1 , specifically including the following:

[0090] Step 100: Acquire a local feature image frame of the image data according to a pre-generated convolutional neural network and image data collected by the robot;

[0091] Step 200: Generate a global descriptor of the image data according to the pre-generated contraction autoencoder and the local feature image frame;

[0092] Step 300: Select a target image frame of the image data according to the global descriptor and a threshold range of a viewing angle change of the robot.

[0093] From the above description, it can be seen that an embodiment of the present invention provides a target image frame selection method suitable for a robot exploration scenario, including: first, obtaining a local feature image frame of image data based on a pre-generated convolutional neural network and image data collected by the robot; then, generating a global descriptor of the image data based on the pre-generated contraction autoencoder and the local feature image frame; finally, selecting a target image frame of the image data based on the global descriptor and the robot's perspective change threshold range.

[0094] Aiming at the problems of robot positioning and mapping accuracy in exploration, development and operation fields, the present invention designs a key frame selection method. A low-dimensional global descriptor is obtained by pre-training CNN and CAE networks, and the average parallax is calculated using the perspective change theory. More robust key frames are selected to participate in the positioning and mapping of the robot system, helping to improve the performance of the robot in exploration, development and operation fields.

[0095] For step 100, a convolutional neural network (CNN) is a deep learning model consisting of a group of interconnected neurons that transmit information through weights. Neural networks can process and analyze large amounts of data through learning and optimization. The basic unit of a neural network is a neuron, also known as a node or perceptron. Each neuron receives input from other neurons and calculates output through an activation function. The connection weights between neurons determine the degree of influence of the input on the output. By adapting these weights, the neural network can learn patterns and relationships from the training data and make predictions on unseen data.

[0096] Neural networks usually consist of multiple layers, including input layers, hidden layers, and output layers. The input layer receives raw data, the hidden layer is used to process and extract features, and the output layer gives the final prediction results. Each layer contains multiple neurons, and the connections between them form a forward propagation network structure. The back propagation algorithm is used to train neural networks. The algorithm calculates the error between the predicted value and the true value and adjusts the weight of each connection according to the error to gradually optimize the model. By repeatedly iterating this process, the neural network can continuously improve its prediction ability.

[0097] The core of CNN is the convolution layer, which extracts local features of the image by convolving the input image with a series of convolution kernels. The convolution operation can capture low-level features such as edges and textures in the image. The convolution layer is usually followed by an activation function, such as ReLU, to introduce nonlinearity.

[0098] In order to reduce the number of parameters in the model, a pooling layer is used here to reduce the size of the feature map. The pooling operation can aggregate the feature values ​​in a region, usually taking the maximum value (max pooling) or the mean (average pooling), thereby reducing the size of the feature map. In addition, the application also uses a fully connected layer to map the feature map to the target category. The neurons in the fully connected layer are connected to all the neurons in the previous layer, and the relationship between the features and the categories is determined by learning weights.

[0099] The descriptor in step 200 is used to describe and represent the numerical value or numerical vector of the specific features (e.g., edge, corner point, texture, etc.) of the current image frame collected by the robot. Specifically, a set of vectors or a set of numerical features extracted from the original data can be used as a descriptor to extract discriminative and repeatable features so as to perform matching, classification, retrieval and other operations in subsequent tasks.

[0100] Preferably, the descriptor includes SIFT (Scale Invariant Feature Transform), SURF (Speeded Up Robust Features), ORB (Rotation Invariant Binary Descriptor), etc. These descriptors usually extract local features with stability and discrimination from the image, such as key points, edge directions, etc.

[0101] Contractive AutoEncoder (CAE) is a special kind of autoencoder. An autoencoder is an unsupervised neural network used to learn an effective representation (or encoding) of data. An autoencoder consists of two parts: an encoder and a decoder. The encoder is responsible for converting the input data (such as an image or text) into a more compact form of expression, while the decoder is used to reconstruct the original input from this compact expression. In this application, the contractive autoencoder adds a constraint on this basis, that is, it adds a regularization term so that the encoded representation is insensitive to small changes in the input. Specifically, the loss function of the contractive autoencoder includes not only the reconstruction error, but also the impact of small changes in the input data on the encoded representation (i.e., the Frobenius norm of the Jacobian matrix of the encoded representation). This constraint encourages the model to learn a robust representation that keeps the encoded representation stable even when there are small changes in the input data. This can make the model more robust and improve the generalization ability of the model.

[0102] For step 300, preferably, the target image frame in step 300 is a key frame, that is, a representative frame among multiple ordinary frames. Extracting the key frame of the image data collected by the robot has the following beneficial effects:

[0103] 1. The information redundancy between adjacent frames is very high. The key frame is to take the most representative frame among the local adjacent frames to reduce the information redundancy.

[0104] 2. The key frame is the result of normal frame flow and optimization, which prevents useless or erroneous information from entering the optimization process and destroying the accuracy of fixed-generation mapping.

[0105] 3. If all frames are involved in the calculation, it will not only waste computing power, but also be a huge test for memory.

[0106] Embodiment 2:

[0107] In some embodiments of the present invention, the convolutional neural network model is a VGG-16 model, see Figure 2 , step 100 comprises:

[0108] Step 101: extracting a descriptor of a current image frame of the image data according to the VGG-16 model;

[0109] The VGG-16 model is a deep convolutional neural network structure consisting of 16 layers of convolutional neural networks, including 13 convolutional layers and 3 fully connected layers. The characteristics of this network structure are clear hierarchy, all convolutional layers use 3x3 convolution kernels, pooling layers use 2x2 pooling kernels, and each convolutional layer is followed by a ReLU activation function.

[0110] Step 102: Acquire the local feature image frame according to the descriptor and the VGG-16 model.

[0111] Specifically, a pre-trained CNN is used to extract descriptors from the image frame captured at time t, and the last layer outputs the result to obtain a local feature image frame.

[0112] Traditional methods widely used in SLAM systems are difficult to apply to large-scale environments such as severe deformation and motion blur. The use of convolutional neural networks (CNNs) can be applied to visual feature extraction in large-scale scenes.

[0113] Specifically, for the image frame I obtained by the visual front end at time t t , using the classification-based pre-trained VGG16 model to extract the descriptor M t To explain the visual features of the scene, we get the local feature image frame M t :

[0114] M t =C θ (I t )

[0115] Among them, C θ represents the pre-trained CNN neural network, M t It is the output result of the last layer of pre-trained CNN;

[0116] In some embodiments of the present invention, see Figure 3 , step 200 comprises:

[0117] Step 201: generating a gradient histogram of the local feature image frame according to the contraction autoencoder;

[0118] Step 202: reconstruct the partial feature image frame according to the contraction autoencoder and the gradient histogram to generate the global descriptor.

[0119] In step 201 and step 202, since the descriptors obtained by the classification-based pre-trained CNN have local features and cannot meet the requirements of global consistency, and the extracted local descriptors have high dimensions, are difficult to adapt to complex environments, and have poor generalization capabilities; CAE has demonstrated superior performance in semantic segmentation, reconstructing gradient histograms, feature encoding, and compressing local feature maps. Using CAE can improve the performance of local descriptors and help obtain more robust global descriptors.

[0120] Specifically, the unsupervised CAE method is used to reconstruct the gradient histogram of the image frame acquired by the visual front end. CAE consists of three encoders and three decoders:

[0121] F θ=[f enco ,f deco ]

[0122] Among them, CAE uses F θ To express, f enco represents the encoder, f deco represents a decoder;

[0123] The gradient histogram is input into the contraction autoencoder, and the encoder output of the CAE method is used as the global descriptor of the low-dimensional mapping expression.

[0124]

[0125] The loss function used is:

[0126]

[0127] Among them, m represents the local feature map M obtained by the pre-trained CNN t During inference, only the encoder is retained to generate the global descriptor.

[0128] In some embodiments of the present invention, see Figure 4 , which is applicable to the target image frame selection method in the robot exploration scene, and also includes:

[0129] Step 400: normalize the global descriptor.

[0130] The global descriptor is normalized to obtain the final global descriptor.

[0131] Specifically, the descriptor output by the CAE encoder is L2 normalized to obtain the final global descriptor M; specifically:

[0132] Normalize the global descriptor vector by dividing it by its L2 norm (the square root of the sum of the squares of the vector elements). The formula is as follows:

[0133] Normalized vector = original vector / L2 norm of original vector.

[0134] The above method can ensure that the direction of the original data of the global descriptor remains unchanged, while avoiding potential problems caused by feature scale differences in the machine learning model.

[0135] In some embodiments of the present invention, see Figure 5 , which is applicable to the target image frame selection method in the robot exploration scene, and also includes:

[0136] Step 500: Generate the convolutional neural network, see Figure 6, step 500 comprises:

[0137] Step 501: Generate training data and verification data according to the history of the image data, and perform the following iterative operations:

[0138] Step 502: training the pre-constructed initial model or training model of the convolutional neural network according to the training data, wherein the training model is generated by training the initial model according to the training data;

[0139] Constructing the initial model of a convolutional neural network includes the following steps: First, determine the structure of the neural network, including the number of layers, the number of neurons in each layer, the activation function, etc. These parameters can be selected according to specific problems and needs. Next, initialize the weights. For the weights of each connection, you can use random numbers to initialize them. Preferably, randomly extract values ​​from a uniform distribution or a Gaussian distribution as the initial weights. This can be implemented using a library function or a custom function. Initialize the bias. For the bias of each neuron, you can also use a random number to initialize it. You can also randomly extract values ​​from a uniform distribution or a Gaussian distribution as the initial bias. Select an activation function: Each neuron in a neural network usually has an activation function to introduce nonlinear characteristics. Common activation functions include sigmoid, ReLU, tanh, etc.

[0140] Step 503: verifying the training model according to the verification data to generate a verification result;

[0141] Step 504: When the verification result reaches a preset training cutoff condition, stop training to generate the convolutional neural network;

[0142] Training cutoff conditions include the following: Fixed number of rounds: Training can be set to stop after reaching a specified number of rounds. This is one of the simplest cutoff conditions, but it may not fully utilize the information of the dataset, resulting in underfitting or overfitting problems. Reach a specified accuracy or error: You can set a target accuracy or error value and stop training when the model reaches or approaches this target. This ensures that the model reaches a certain performance level. Early stopping: You can monitor the performance of the validation set and stop training when the performance no longer improves. You can set a tolerance (such as no performance improvement for multiple consecutive rounds) to determine when to stop training. Computational resource constraints: Training time may be limited by computing resources, such as time, memory, or computing power. You can set a time limit or resource limit and stop training when this limit is exceeded. Cross-validation: You can use cross-validation to evaluate the performance of the model and stop training when the cross-validation error no longer decreases.

[0143] In some embodiments of the present invention, the contraction autoencoder consists of 3 sub-encoders and 3 sub-decoders.

[0144] It can be understood that by reconstructing the local feature image frame using the CAE composed of three encoders and three decoders, a low-dimensional global descriptor can be obtained.

[0145] In some embodiments of the present invention, see Figure 7 , step 300 comprises:

[0146] Step 301: generating disparity between adjacent image frames according to the global descriptor;

[0147] Specifically, by setting the robot's perspective change threshold, the average disparity between adjacent image frames is calculated based on the global descriptor.

[0148] Furthermore, according to the local descriptors obtained from the last layer of the pre-trained CNN, after CAE reconstructed the local feature map histogram and performed local descriptor dimensionality reduction, a robust global descriptor was obtained through a normalization process; the key frame was selected by combining the extracted global descriptor with the viewpoint change: the average disparity of adjacent image frames was calculated using the obtained global descriptor;

[0149]

[0150] in, It refers to the global descriptor set output by CAE of the image frame at time t1. It refers to the set of global descriptors output by CAE of the image frames at the adjacent time t2; n represents the number of global descriptors obtained;

[0151] Step 302: selecting a candidate image frame of the target image frame according to the disparity and the viewing angle change threshold range;

[0152] Step 303: in the candidate image frame, determining the candidate score of the current image frame according to the number of inliers of the current image frame and the image frame before the current image frame;

[0153] Step 304: Select the target image frame from the candidate image frames according to the candidate scores.

[0154] In step 302 to step 304, the image frames that meet the perspective change threshold range are regarded as candidate key frames, and the candidate scores are calculated based on the key points extracted from the current candidate key frame and the number of inliers with the previous key frame, and the candidate frame with the highest score is selected as the key frame.

[0155] For image frames that meet the viewing angle change threshold range, the number of key points k extracted from the current image frame is calculated, and the number of inliers s between the current image frame and the previous key frame is calculated by PnP and Ransac algorithms. The candidate score of the i-th image frame that meets the viewing angle change threshold range is obtained as follows:

[0156]

[0157] Wherein, L represents the number of candidate image frames that meet the viewing angle change threshold range, and the candidate image frame with the highest score is selected as the key frame after normalization processing;

[0158] In some embodiments of the present invention, see Figure 8 , which is applicable to the target image frame selection method in the robot exploration scene, and also includes:

[0159] Step 600: determining an acceleration threshold of the robot according to the topographic data of the exploration block where the robot is located and the acquisition requirement data;

[0160] Step 700: When the acceleration of the robot exceeds the acceleration threshold, reselect the target image frame.

[0161] In step 600 and step 700, an acceleration threshold is set, and a new key frame is selected when the motion state exceeds the acceleration threshold. Specifically, in order to deal with the image blur problem caused by rapid motion, a second sensor IMU is added to track the real-time state of the robot system. This process is independent of the vision-based key frame selection process: an acceleration threshold ε is set, and a new key frame is selected when the acceleration of the system exceeds the threshold.

[0162] From the above description, it can be seen that an embodiment of the present invention provides a target image frame selection method suitable for a robot exploration scenario, including: first, obtaining a local feature image frame of image data based on a pre-generated convolutional neural network and image data collected by the robot; then, generating a global descriptor of the image data based on the pre-generated contraction autoencoder and the local feature image frame; finally, selecting a target image frame of the image data based on the global descriptor and the robot's perspective change threshold range.

[0163] Specifically, a pre-trained CNN is first used to extract descriptors for the image frames captured at time t, and the results are output by the last layer of CNN to obtain local feature image frames; then, the local feature image frames are reconstructed using a CAE consisting of three encoders and three decoders to obtain a low-dimensional global descriptor; the global descriptor is normalized to obtain the final global descriptor; a perspective change threshold is set, and the average disparity between adjacent image frames is calculated based on the global descriptor; image frames that meet the perspective change threshold range are regarded as candidate key frames, and the candidate scores are calculated based on the key points extracted from the current candidate key frame and the number of inliers with the previous key frame, and the candidate frame with the highest score is selected as the key frame; finally, an acceleration threshold is set, and a new key frame is selected when the motion state exceeds the acceleration threshold.

[0164] Aiming at the problems of robot positioning and mapping accuracy in exploration, development and operation fields, the present invention designs a key frame selection method. A low-dimensional global descriptor is obtained by pre-training CNN and CAE networks, and the average parallax is calculated using the perspective change theory. More robust key frames are selected to participate in the positioning and mapping of the robot system, helping to improve the performance of the robot in exploration, development and operation fields.

[0165] Embodiment three:

[0166] In a specific embodiment, the present invention also provides a specific embodiment of a target image frame selection method suitable for a robot exploration scene, see Fig. 9 as well as Fig.10 , specifically including the following steps.

[0167] S1: Use the pre-trained CNN to extract the descriptor of the image frame captured at time t, and the last layer outputs the result to obtain the local feature image frame;

[0168] S2: Reconstruct the local feature image frame using a CAE consisting of three encoders and three decoders to obtain a low-dimensional global descriptor;

[0169] S3: normalize the global descriptor to obtain the final global descriptor;

[0170] See also Fig.11 , perform L2 normalization on the descriptor output by the CAE encoder to obtain the final global descriptor.

[0171] S4: setting a perspective change threshold and calculating the average disparity between adjacent image frames based on the global descriptor;

[0172] S5: consider the image frames that meet the view change threshold range as candidate key frames, calculate the candidate score according to the key points extracted from the current candidate key frame and the number of inliers with the previous key frame, and select the candidate frame with the highest score as the key frame;

[0173] S6: Set the acceleration threshold. When the motion state exceeds the acceleration threshold, select a new key frame.

[0174] Aiming at the problems of robot positioning and mapping accuracy in exploration, development and operation fields, the present invention designs a key frame selection method. A low-dimensional global descriptor is obtained by pre-training CNN and CAE networks, and the average parallax is calculated using the perspective change theory. More robust key frames are selected to participate in the positioning and mapping of the robot system, helping to improve the performance of the robot in exploration, development and operation fields.

[0175] Embodiment 4:

[0176] Based on the same inventive concept, the embodiments of the present application also provide a target image frame selection device suitable for robot exploration scenarios, which can be used to implement the methods described in the above embodiments, such as the following embodiments. Since the principle of solving the problem by the target image frame selection device suitable for robot exploration scenarios is similar to that of the target image frame selection method suitable for robot exploration scenarios, the implementation of the target image frame selection device suitable for robot exploration scenarios can refer to the implementation of the target image frame selection method suitable for robot exploration scenarios, and the repeated parts will not be repeated. As used below, the terms "unit" or "module" can be a combination of software and / or hardware that implements predetermined functions. Although the system described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0177] The embodiment of the present invention provides a specific implementation of a target image frame selection device suitable for a robot exploration scene that can implement a target image frame selection method suitable for a robot exploration scene, see Fig.12 , a target image frame selection device suitable for robot exploration scenarios includes:

[0178] A local feature image frame acquisition module 10, used to acquire a local feature image frame of the image data according to the image data collected by the robot and a pre-generated convolutional neural network;

[0179] A global descriptor generation module 20, configured to generate a global descriptor of the image data according to a pre-generated contraction autoencoder and the local feature image frame;

[0180] The target image frame selection module 30 is used to select the target image frame of the image data according to the global descriptor and the viewing angle change threshold range of the robot.

[0181] In one embodiment of the present invention, the convolutional neural network model is a VGG-16 model, and the local feature image frame acquisition module includes:

[0182] A descriptor extraction unit, configured to extract a descriptor of a current image frame of the image data according to the VGG-16 model;

[0183] A local feature image frame acquisition unit is used to acquire the local feature image frame according to the descriptor and the VGG-16 model.

[0184] In one embodiment of the present invention, the global descriptor generation module includes:

[0185] A gradient histogram generating unit, configured to generate a gradient histogram of the local feature image frame according to the contraction autoencoder;

[0186] A global descriptor generating unit is used to reconstruct the local feature image frame according to the contraction autoencoder and the gradient histogram to generate the global descriptor.

[0187] In one embodiment of the present invention, a target image frame selection device suitable for a robot exploration scene also includes:

[0188] The global descriptor normalization module is used to perform normalization processing on the global descriptor.

[0189] In one embodiment of the present invention, the target image frame selection device applicable to the robot exploration scene further includes: a convolutional neural network generation module, which is used to generate the convolutional neural network, and the convolutional neural network generation module includes:

[0190] The training data generating unit is used to generate training data and verification data according to the history of the image data, and perform the following iterative operations:

[0191] A model training unit, configured to train a pre-constructed initial model or a training model of the convolutional neural network according to the training data, wherein the training model is generated by training the initial model according to the training data;

[0192] A verification result generating unit, used to verify the training model according to the verification data to generate a verification result;

[0193] A convolutional neural network generation unit, configured to stop training when the verification result reaches a preset training cutoff condition to generate the convolutional neural network;

[0194] The contraction autoencoder consists of 3 sub-encoders and 3 sub-decoders.

[0195] In one embodiment of the present invention, the target image frame selection module includes:

[0196] a disparity generating unit, configured to generate disparity between adjacent image frames according to the global descriptor;

[0197] A candidate image frame selection unit, configured to select a candidate image frame of a target image frame according to the disparity and the viewing angle change threshold range;

[0198] A candidate score determination unit, configured to determine, within the candidate image frame, a candidate score of the current image frame according to the number of inliers of the current image frame and an image frame preceding the current image frame;

[0199] A target image frame selection unit is used to select the target image frame from the candidate image frames according to the candidate scores.

[0200] In one embodiment of the present invention, a target image frame selection device suitable for a robot exploration scene also includes:

[0201] An acceleration threshold setting module, used to determine the acceleration threshold of the robot according to the topographic data of the exploration block where the robot is located and the acquisition demand data;

[0202] The target image frame reselection module is used to reselect the target image frame when the acceleration of the robot exceeds the acceleration threshold.

[0203] From the above description, it can be seen that an embodiment of the present invention provides a target image frame selection device suitable for a robot exploration scenario, including: a local feature image frame acquisition module, used to acquire local feature image frames of image data based on a pre-generated convolutional neural network and image data collected by the robot; a global descriptor generation module, used to generate a global descriptor of image data based on a pre-generated contraction autoencoder and the local feature image frame; a target image frame selection module, used to select a target image frame of image data based on the global descriptor and the robot's perspective change threshold range.

[0204] The target image frame selection device suitable for robot exploration scenarios provided in the embodiments of the present invention can better improve the performance of robots in the fields of exploration, development and operations, and provide a key frame selection device for robots applied to visual inertial SLAM, which can effectively improve the performance of multi-sensor robots in the field of exploration operations.

[0205] Embodiment five:

[0206] The embodiment of the present application also provides a specific implementation of an electronic device capable of implementing all steps in the target image frame selection method applicable to a robot exploration scenario in the above embodiment, see Fig.13 , electronic equipment specifically includes the following:

[0207] Processor (processor) 1201, memory (memory) 1202, communication interface (CommunicationsInterface) 1203 and bus 1204;

[0208] The processor 1201, the memory 1202, and the communication interface 1203 communicate with each other through the bus 1204; the communication interface 1203 is used to realize information transmission between the server device and the client device and other related devices;

[0209] The processor 1201 is used to call the computer program in the memory 1202. When the processor executes the computer program, all the steps in the target image frame selection method applicable to the robot exploration scene in the above embodiment are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0210] Step 100: Acquire a local feature image frame of the image data according to a pre-generated convolutional neural network and image data collected by the robot;

[0211] Step 200: Generate a global descriptor of the image data according to the pre-generated contraction autoencoder and the local feature image frame;

[0212] Step 300: Select a target image frame of the image data according to the global descriptor and a threshold range of a viewing angle change of the robot.

[0213] Embodiment six:

[0214] The embodiments of the present application also provide a computer-readable storage medium capable of implementing all the steps of the target image frame selection method applicable to the robot exploration scenario in the above embodiment. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, all the steps of the target image frame selection method applicable to the robot exploration scenario in the above embodiment are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0215] Step 100: Acquire a local feature image frame of the image data according to a pre-generated convolutional neural network and image data collected by the robot;

[0216] Step 200: Generate a global descriptor of the image data according to the pre-generated contraction autoencoder and the local feature image frame;

[0217] Step 300: Select a target image frame of the image data according to the global descriptor and a threshold range of a viewing angle change of the robot.

[0218] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the hardware + program embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0219] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0220] Although the present application provides method operation steps such as embodiments or flow charts, more or fewer operation steps may be included based on conventional or non-creative labor. The order of steps listed in the embodiments is only one way of executing the order of many steps and does not represent the only execution order. When the actual device or client product is executed, it can be executed in the order of the method shown in the embodiments or the drawings or in parallel (for example, in a parallel processor or multi-threaded processing environment).

[0221] For the convenience of description, the above devices are described in various modules according to their functions. Of course, when implementing the embodiments of this specification, the functions of each module can be implemented in the same or more software and / or hardware, or the module implementing the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0222] Those skilled in the art also know that, in addition to implementing the controller in a purely computer-readable program code, the controller can be made to implement the same function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered as a hardware component, and the devices for implementing various functions included therein can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules for implementing the method and structures within the hardware component.

[0223] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0224] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0225] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. In the description of this specification, the description of the reference terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of this specification. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, in the absence of contradiction, a person skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0226] The above is only an example of the embodiment of the present specification and is not intended to limit the embodiment of the present specification. For those skilled in the art, the embodiment of the present specification may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the embodiment of the present specification shall be included in the scope of the claims of the embodiment of the present specification.

Claims

1. A target image frame selection method suitable for robot exploration scenarios, characterized in that: include: Acquire a local feature image frame of the image data according to the image data collected by the robot and a pre-generated convolutional neural network; Generate a global descriptor of the image data based on the pre-generated contraction autoencoder and the local feature image frame; A target image frame of the image data is selected according to the global descriptor and a threshold range of a viewing angle change of the robot.

2. The target image frame selection method according to claim 1, characterized in that: The convolutional neural network model is a VGG-16 model, and the local feature image frame of the image data is obtained according to the image data collected by the robot and the pre-generated convolutional neural network, including: Extracting a descriptor of a current image frame of the image data according to the VGG-16 model; The local feature image frame is acquired according to the descriptor and the VGG-16 model.

3. The target image frame selection method according to claim 1, characterized in that: Generating a global descriptor of the image data according to the pre-generated contraction autoencoder and the local feature image frame, comprising: Generating a gradient histogram of the local feature image frame according to the contraction autoencoder; The partial feature image frame is reconstructed according to the contraction autoencoder and the gradient histogram to generate the global descriptor.

4. The target image frame selection method according to claim 3, characterized in that: Also includes: The global descriptor is normalized.

5. The target image frame selection method according to claim 1, characterized in that: The steps of generating the convolutional neural network include: Generate training data and verification data based on the history of the image data, and perform the following iterative operations: Training a pre-constructed initial model or training model of the convolutional neural network according to the training data, wherein the training model is generated by training the initial model according to the training data; Verifying the training model according to the verification data to generate a verification result; When the verification result reaches a preset training cutoff condition, the training is stopped to generate the convolutional neural network; The contraction autoencoder consists of 3 sub-encoders and 3 sub-decoders.

6. The target image frame selection method according to any one of claims 1 to 5, characterized in that: Selecting a target image frame of the image data according to the global descriptor and a threshold range of a viewing angle change of the robot includes: generating disparity between adjacent image frames according to the global descriptor; Selecting a candidate image frame of the target image frame according to the disparity and the viewing angle change threshold range; In the candidate image frame, determining a candidate score of the current image frame according to the number of inliers of the current image frame and an image frame previous to the current image frame; The target image frame is selected from the candidate image frames according to the candidate scores.

7. The target image frame selection method according to any one of claims 1 to 5, characterized in that: Also includes: Determining an acceleration threshold of the robot according to geomorphic data of the exploration block where the robot is located and acquisition requirement data; When the acceleration of the robot exceeds the acceleration threshold, the target image frame is reselected.

8. A target image frame selection device suitable for robot exploration scenes, characterized in that: include: A local feature image frame acquisition module, used to acquire a local feature image frame of the image data based on the image data collected by the robot and a pre-generated convolutional neural network; A global descriptor generation module, used to generate a global descriptor of the image data according to the pre-generated contraction autoencoder and the local feature image frame; The target image frame selection module is used to select the target image frame of the image data according to the global descriptor and the viewing angle change threshold range of the robot.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the target image frame selection method applicable to a robot exploration scenario as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the target image frame selection method applicable to a robot exploration scene as described in any one of claims 1 to 7 are implemented.

Images