A sequence image scene recognition method based on deep learning

Abstract

The application relates to a sequence image scene recognition method based on deep learning, which comprises the following steps: acquiring a current observation image and an image sequence of a map, and extracting a descriptor of the image; calculating a sequence value between the current observation image and the map according to the extracted descriptor; estimating a current speed value based on the sequence value, and determining a current likelihood probability distribution; determining a confidence degree of a current state based on the current likelihood probability distribution and a prediction confidence degree of a previous state; determining a probability distribution of state transition based on the current speed value; and determining a prediction confidence degree of a next state based on the probability distribution of state transition and the confidence degree of the current state. According to the scene recognition scheme, the matching error of a single picture is overcome through matching between sequence images; and the robot can realize positioning function only through a visual sensor in the case that the outdoor scene changes with seasons, weather and time.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to a method for scene recognition of sequential images based on deep learning. Background Technology

[0002] Image scene recognition refers to the automatic processing and analysis of images using their visual information to identify specific scenes (such as kitchens, streets, mountains, etc.) within them. Identifying the scene in an image not only aids in understanding its overall semantic content but also provides a basis for recognizing specific targets and events within the image. Therefore, scene recognition plays a crucial role in computer-aided image understanding. Scene recognition technology can be applied to many practical problems, such as intelligent image management and retrieval.

[0003] For recognizing changing scenes, matching single images has a significant error rate. Therefore, many recognition methods focus on adjusting the system's recognition approach to improve robustness. Many machine learning-based methods are also used to learn how scenes change; these methods are based on the assumption that the appearance of scenes at different locations can transform into each other over time, possessing generality and learnability. This assumption has been validated in some studies.

[0004] In current robot localization methods, GPS + LiDAR is mainly used to achieve real-time positioning of autonomous vehicles. This approach is costly and provides less information than vision-based methods. A major challenge in scene recognition-based localization lies in effectively identifying changing scenes, for which there is currently no mature solution. Summary of the Invention

[0005] To at least partially overcome the problems existing in related technologies, this application provides a deep learning-based method for scene recognition of sequential images.

[0006] According to a first aspect of the embodiments of this application, a deep learning-based sequence image scene recognition method is provided, comprising:

[0007] Obtain the image sequence of the currently observed image and map, and extract the image descriptors;

[0008] Calculate the sequence cost between the currently observed image and the map based on the extracted descriptors;

[0009] Estimate the current velocity value based on the sequence cost and determine the current likelihood probability distribution;

[0010] Determine the confidence level of the current state based on the current likelihood probability distribution and the prediction confidence level of the previous state.

[0011] Determine the probability distribution of state transitions based on the current velocity value;

[0012] Based on the probability distribution of state transitions and the confidence level of the current state, the prediction confidence level of the next state is determined.

[0013] Further, the extracted image descriptor includes:

[0014] Images are processed using a convolutional neural network model, with the output of a specified layer used as a descriptor; wherein the convolutional neural network model is a model trained on a scene classification dataset;

[0015] The descriptors are normalized.

[0016] Further, the sequential cost value between the currently observed image and the map is calculated, including:

[0017] Let the set of currently observed images observed by the agent be denoted as: Q = [I1 I2 ... I j ... I n The image sequence of a map is denoted as: M = [I1 I2 ... I i ... I m ];

[0018] Image I i With Image I j The sequence cost value between images I is: i Descriptors and Image I j The value of the cosine distance between the descriptors; where I i ∈M,I j ∈Q.

[0019] Further, estimating the current velocity value based on the sequence cost includes:

[0020] Based on the current time, determine a search sequence of a preset length;

[0021] The summation of sequence values ​​for the entire search sequence is used as the degree of matching between the currently observed image and the map.

[0022] By adjusting the slope v of the search sequence, the search sequence z with the highest matching degree is selected. T ;

[0023] Search sequence z T The corresponding slope v is determined as the agent's current velocity value.

[0024] Further, determining the current likelihood probability distribution includes:

[0025] Using the search sequence zT Based on the baseline, select N candidate sequences from the vicinity; where N is a preset value;

[0026] Calculate the probability distribution of N candidate sequences;

[0027] Assuming the obtained probability distribution is a sample of a Gaussian distribution, the parameters of the Gaussian distribution are estimated based on the obtained probability distribution, and the estimated Gaussian distribution is used as the current likelihood probability distribution.

[0028] Furthermore, based on the current likelihood probability distribution and the prediction confidence of the previous state, the confidence of the current state is determined, specifically including:

[0029] Based on the current likelihood probability distribution, the prediction confidence of the previous state is recursively calculated using the Kalman filter method to obtain the confidence of the current state.

[0030] Furthermore, after determining the confidence level of the current state, the following steps are also included:

[0031] Based on the confidence level of the current state, determine the recognition result of the currently observed image.

[0032] Furthermore, the probability distribution of state transitions is determined based on the current velocity value, including:

[0033] Assume the probability distribution of state transitions follows a Gaussian distribution;

[0034] Based on the current velocity value, the dynamic equations describing the state transition and the equations predicting the process are solved to obtain the probability distribution of the state transition.

[0035] Furthermore, based on the probability distribution of state transitions and the confidence level of the current state, the prediction confidence level of the next state is determined, including:

[0036] Based on the probability distribution of state transitions, the confidence of the current state is recursively calculated using Kalman filtering to obtain the predicted confidence of the next state.

[0037] According to a second aspect of the embodiments of this application, a sequence image scene recognition device based on deep learning is provided, comprising:

[0038] The acquisition module acquires image sequences of the currently observed image and the map.

[0039] The extraction module is used to extract image descriptors;

[0040] The cost value calculation module is used to calculate the sequential cost value between the currently observed image and the map based on the extracted descriptors;

[0041] The estimation module is used to estimate the current velocity value based on the sequence cost.

[0042] The first probability determination module is used to determine the current likelihood probability distribution;

[0043] The update module is used to determine the confidence level of the current state based on the current likelihood probability distribution and the prediction confidence level of the previous state.

[0044] The second probability determination module is used to determine the probability distribution of state transitions based on the current velocity value;

[0045] The prediction module is used to determine the prediction confidence of the next state based on the probability distribution of state transitions and the confidence of the current state.

[0046] The technical solutions provided by the embodiments of this application have the following beneficial effects:

[0047] This invention provides a deep learning-based sequence image scene recognition scheme, which overcomes the problem of large matching errors when matching single images by matching between sequence images. It enables robots to perceive their surroundings and achieve localization even when outdoor scenes change with seasons, weather, and time, using only visual sensors. This lays the foundation for further scene recognition-based localization and navigation.

[0048] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0049] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0050] Figure 1 This is a flowchart illustrating a deep learning-based sequence image scene recognition method according to an exemplary embodiment.

[0051] Figure 2 This is a system flowchart shown in an embodiment of this application.

[0052] Figure 3 This is a schematic diagram of sequence cost matching shown in an embodiment of this application.

[0053] Figure 4 This is a schematic diagram of an implicit Markov chain shown in an embodiment of this application. Detailed Implementation

[0054] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of methods and apparatus consistent with some aspects of this application as detailed in the appended claims.

[0055] To further elaborate on the technical solution of this application, the current state of the technology in this field will be introduced first.

[0056] Scene recognition is a highly challenging problem. Whether for humans, animals, robots, or autonomous vehicles, achieving scene recognition requires certain basic conditions: First, a reference "map" is needed. This map isn't a typical everyday map or a point cloud map used in autonomous driving, but rather a continuous sequence of images or a few scattered images of landmark locations. It represents prior information, known facts, and serves as the basis for recognition. Second, the system should be able to convert the current image into a corresponding location confidence score. This score indicates whether the image is within our map, and if so, its specific location.

[0057] Therefore, the scene recognition problem can be divided into three parts: First, a good scene description method is needed to abstract the rich and huge two-dimensional image information of the scene into a simpler mathematical description, i.e., a descriptor; second, this scene information needs to be normalized into map information, that is, the geometric logical relationship between scenes needs to be established; third, an effective method is needed to convert the current scene state of the agent into the location confidence of the scene in the map.

[0058] Current scene recognition methods can be divided into extracting local descriptors and extracting global descriptors.

[0059] (1) Local Descriptors. Many studies have focused on traditional local feature descriptors, such as SIFT and SURF, to explore their robustness in changing scenarios. Furgale, Barfoot, and others pointed out in their research that the traditional SURF descriptor is not robust to changes in illumination. Valgren, Lilienthal, and others pointed out in their research that various variants of SURF and SIFT do not have strong robustness under conditions of illumination, fog, and seasonal changes.

[0060] (2) Global Descriptors. Besides local descriptors, many studies are also examining other forms of scene description. For example, SeqSLAM and its variants both use descriptions of the entire image to achieve robustness. Convolutional neural networks also play a significant role here. Many studies have shown that convolutional neural networks can learn more generalized features, which are applicable to many related visual tasks. That is, descriptors trained for object and scene classification can also be used for scene recognition tasks. Sunderhauf et al.'s research indicates that descriptors generated by intermediate layers of convolutional neural networks, such as convolutional layers, are highly robust to changes in scene appearance, while higher layers, such as fully connected layers, are highly robust to changes in viewpoint. Grag et al.'s research suggests that normalizing descriptors from fully connected layers can significantly improve their robustness to appearance changes. Dai et al. compared CNN descriptors with traditional descriptors, finding that CNN descriptors are more robust through experiments on datasets with different viewpoints and lighting conditions.

[0061] Figure 1 This is a flowchart illustrating a deep learning-based sequence image scene recognition method according to an exemplary embodiment. The method may include the following steps:

[0062] Step S1: Obtain the image sequence of the currently observed image and the map, and extract the image descriptors; wherein, the "map" includes a continuous image sequence or images of multiple landmark locations;

[0063] Step S2: Calculate the sequence cost between the currently observed image and the map based on the extracted descriptors;

[0064] Step S3: Estimate the current velocity value based on the sequence cost and determine the current likelihood probability distribution;

[0065] Step S4: Determine the confidence level of the current state based on the current likelihood probability distribution and the prediction confidence level of the previous state;

[0066] Step S5: Determine the probability distribution of state transition based on the current velocity value;

[0067] Step S6: Based on the probability distribution of state transitions and the confidence level of the current state, determine the prediction confidence level of the next state.

[0068] This invention provides a deep learning-based sequence image scene recognition scheme, which overcomes the problem of large matching errors when matching single images by matching between sequence images. It enables robots to perceive their surroundings and achieve localization even when outdoor scenes change with seasons, weather, and time, using only visual sensors. This lays the foundation for further scene recognition-based localization and navigation.

[0069] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0070] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings.

[0071] The process of this identification method is as follows: Figure 2 As shown, x t v represents the current state of the agent. t The current speed of the agent, M represents the sequence of map images referenced by the agent, bel t and This represents the agent's current confidence level at time t and its predicted confidence level at time t+1. The latest recognition result is obtained continuously through recursion.

[0072] 1) The first step is to obtain the descriptor of the current image through a convolutional neural network. Using the observed image information of the current agent and the image in the map, its sequence cost is calculated. Simultaneously, the current velocity value is estimated using the sequence cost. Obtain the observed likelihood probability distribution p(z) t |x t ,M);

[0073] 2) The second step is to use the observed likelihood probability distribution p(z) t |x t (M) and the confidence level of the previous stage prediction bel pred (x t To update the current probability distribution and obtain the confidence level of the current state, bel(x). t (Algorithm 3-1, lines 18-20);

[0074] 3) The third step is to estimate the current speed. To obtain the probability distribution of state transitions And by combining it with the confidence level of the current state, bel(x) t To predict the confidence level of the next stage state. pred(x t+1 ).

[0075] This method mimics the mechanism by which humans can navigate by abstracting and understanding complex scenes to obtain the most basic scene information. It designs a robust scene recognition system based on deep learning scene description, combined with reasoning and decision-making, and realizes localization and navigation based on this scene recognition method.

[0076] The specific algorithm of this invention is shown in the table below:

[0077]

[0078]

[0079] The algorithm proposed in this invention will be described in detail below with reference to specific embodiments.

[0080] I. Normalization of Image Description in Convolutional Neural Networks

[0081] First, step S1 is executed to obtain the image sequence of the currently observed image and the map, and to extract the image descriptors. In some embodiments, the step of extracting descriptors includes: processing the image using a convolutional neural network model, using the output of a specified layer as the descriptor; and normalizing the descriptor. The convolutional neural network model is a model trained on a scene classification dataset.

[0082] Classic convolutional neural network models, such as AlexNet, are trained using scene classification datasets, such as Place365, and the outputs of certain layers are used as image descriptions, such as the output of the fc6 layer containing 4096 variables. A descriptor normalization method is employed to significantly improve the robustness of fully connected layer descriptors to appearance changes.

[0083]

[0084] in, The output of the feature descriptor at layer fc6 representing image i, while and This represents the mean and variance of the image descriptors in the entire map database. And as a result... This is the normalized descriptor, called the Normalized Set of Descriptors (NSD). Here, we also use such a normalized deep feature descriptor as the scene description method for this approach.

[0085] II. Sequence Cost Matching and Related Probability Model Construction

[0086] Having a robust scene description, we should also analyze how to effectively utilize descriptor information to achieve robust matching. To improve matching accuracy, we shouldn't only consider matching a single image; for a scene, we should comprehensively consider the scenes at its preceding locations.

[0087] Next, step S2 is performed, which calculates the sequence cost between the currently observed image and the map based on the extracted descriptors. Step S2 specifically includes:

[0088] Let the set of currently observed images observed by the agent be denoted as: Q = [I1 I2 ... I j ... I n The image sequence of a map is denoted as: M = [I1 I2 ... I i ... I m ];

[0089] Image I i With Image I j The sequence cost value between images I is: i Descriptors and Image I j The value of the cosine distance between the descriptors; where I i ∈M,I j ∈Q.

[0090] 2.1 Construction of the cost matrix

[0091] First, a cost matrix D needs to be constructed, which represents the degree of matching between the agent's perspective and the map image.

[0092] Here, the set of images from the agent's perspective is denoted as Q, where Q = [I1 I2 ... I j ... I n The set of images in the map is denoted as M, where M = [I1 I2 ... I...]. i ... I m ].

[0093] The cost matrix is ​​constructed as shown in Formula 2:

[0094]

[0095] Here, the element D of the matrix i,j For image I in the map i ∈M descriptor and image I from the perspective of the agent j The cosine distance ∈Q describes the similarity between two feature vectors. This cosine distance describes the similarity between two feature vectors and is unaffected by differences in vector magnitudes. Its calculation formula is shown in Formula 3.

[0096]

[0097] in, For image I i ∈M,I j Normalized fc6 descriptor ∈Q.

[0098] 2.2 Sequence Matching and Velocity Estimation

[0099] Next, step S3 is performed to estimate the current velocity value based on the sequence cost value. In some embodiments, estimating the current velocity value based on the sequence cost value specifically includes: determining a search sequence of a preset length based on the current time; summing the sequence costs of the entire search sequence as the degree of matching between the currently observed image and the map; and selecting the search sequence z with the highest degree of matching by adjusting the slope v of the search sequence. T ; Search sequence z T The corresponding slope v is determined as the agent's current velocity value.

[0100] Sequence matching requires searching within a sequence to obtain the best match. Here, we introduce a search space M:

[0101]

[0102] Here, T represents the timestamp of the current moment. The goal is to find a search sequence of length ds to synthesize the matching degree of these ds images, thereby improving the robustness of the recognition. The degree of matching is measured by a sequence cost value.

[0103] Where k = j + v(tT). (5)

[0104] The cost of this sequence is determined by three parameters: the current time point T, the selected frame j in the map, and the matching slope v, such as... Figure 3 As shown.

[0105] Therefore, the cosine distance of the descriptors is used here to represent the matching degree between two images. The sum of the cosine distances over this time period represents the matching degree between the current image and the map. The result with the smallest cosine distance and the highest matching degree is selected by adjusting the slope v. The slope v corresponding to the best result is the current speed of the agent estimated by this scheme, as shown in formula (6). Here, a range V = [v] is set for the speed v. min ,v max This is used to impose a constraint on the optimal estimate to prevent unrealistic values ​​from being generated.

[0106]

[0107] In the formula, D k,t S represents the cosine distance between the image obtained by the agent at time t and image k in the map. T,j This is the matching cost between the image acquired by the agent at time T and image j in the map, as well as previous sequences, used to measure the similarity between the current location and the location in the map. After obtaining the best matching result by changing the value of v, the result is... This scheme estimates the current speed of the agent and uses this estimate as a basis to predict the agent's position at the next time step T+1. Therefore, the optimal matching result is... This is the output result observed at time T. Here, this scheme uses a probability distribution of state transitions. This is used to represent the estimate for the next moment. Here, this scheme makes an important assumption: that the probability distribution follows a Gaussian distribution, i.e., p(x) T+1 |x T ,v esti ,M)~N(μ,σ).

[0108] 2.3 Obtaining the probability distribution

[0109] Next, step S3 is performed to determine the current likelihood probability distribution. In some embodiments, determining the current likelihood probability distribution specifically includes: using the search sequence z T Based on the baseline, select N candidate sequences in the vicinity; where N is a preset value; calculate the probability distribution of the N candidate sequences; assuming that the obtained probability distribution is a sample of the Gaussian distribution, estimate the parameters of the Gaussian distribution based on the obtained probability distribution, and use the estimated Gaussian distribution as the current likelihood probability distribution.

[0110] After obtaining the sequence matching results, we will discuss how to transform this value into a probability distribution, namely the likelihood distribution p(z). T |x T The values ​​are denoted as M, to facilitate the use of probability-based Markov localization in this scheme. Here, the SoftMax function is used to map these values ​​to probabilities ranging from [0,1], as shown in formula (7):

[0111]

[0112] Here S T,k For the image I acquired by the agent at time T T Image I in the map j The cost of the matched sequence. To simplify this operation, this scheme uses a sliding window of length N, W = [zT-N / 2, zT+N / 2], to calculate the probability distribution among the N candidate frames near the optimal matching result:

[0113]

[0114] After converting the cost into a probability, this scheme further assumes that this probability follows a Gaussian distribution, i.e., p(z) T |x T ,M)~N(μo,σo). Here, the least squares method, which has the highest computational efficiency, is used to estimate this Gaussian distribution. Assuming the obtained probability distribution is a sample of a Gaussian distribution, let (x,M)~N(μo,σo). i ,y i If (i = 1, 2, ..., n) is represented, then the following relationship will exist:

[0115]

[0116] Here, the Gaussian distribution has a particularly good property: it is an exponential function of e. We can take its logarithm to transform the exponential form into a linear relationship, and then solve it as a system of linear equations, which can greatly simplify the estimation process of this scheme.

[0117]

[0118] Here, we perform a variable substitution, letting ln(y) i )=z i , The formula would then become a linear equation in the following form:

[0119]

[0120] According to the least squares method, this linear equation can yield an estimate of B:

[0121] B = (X) T X) -1 X T Z (12)

[0122] Therefore, the mean μ, variance σ, and parameter θ can all be obtained from matrix B.

[0123] III. Kalman Filtering – Markov Localization under Gaussian Distribution

[0124] This paper uses Markov positioning to model some of the reasoning behaviors of humans during the localization process. In navigation and localization, humans not only estimate their current state based on current observations but also incorporate their memories, i.e., prior information. Here, this scheme uses an implicit Markov chain to represent this process, such as... Figure 4 As shown.

[0125] The Markov localization process is divided into two sub-processes: prediction and update. The prediction process estimates the agent's state at the next moment by using the state at the previous moment and the state transition model, while the update process uses the predicted state confidence and the observed likelihood probability distribution to update the agent's confidence in its own state at the current moment.

[0126] Here, we assume that the state transitions and observations are linear relationships, and add Gaussian noise, which is a Gaussian linear system. Therefore, this scheme uses Kalman filtering to recursively calculate the agent's confidence in its own state.

[0127] 3.1 Prediction

[0128] Next, step S4 is executed to determine the confidence level of the current state based on the current likelihood probability distribution and the prediction confidence level of the previous state. In some embodiments, step S4 specifically includes: recursively calculating the prediction confidence level of the previous state using a Kalman filter method based on the current likelihood probability distribution to obtain the confidence level of the current state.

[0129] First, the agent's state x i It is composed of map M = {Image i The control commands u of the agent are described by the index value i of the image in the image. i The speed is obtained from estimation. If we describe the state transition, then the dynamic equation describing the state transition is described by the following formula:

[0130]

[0131] This consists of a linear relationship and Gaussian noise, and the prediction process is generally described by the following formula:

[0132]

[0133] In the formula To predict the confidence level, corresponding Figure 2 In As mentioned earlier, the probability distribution describing the state transition It follows a Gaussian distribution, and the confidence level bel(x) i-1 )~N(μ i-1 ,σ i-1 The distribution is also Gaussian, so this scheme can recursively calculate the predicted confidence level bel(x) using Kalman filtering. i )~N(μ i ,σ iIt is also a Gaussian distribution, and its parameters can be calculated using the following equation:

[0134]

[0135] 3.2 Update

[0136] Next, step S5 is executed to determine the probability distribution of the state transition based on the current velocity value. In some embodiments, step S5 specifically includes: assuming that the probability distribution of the state transition is a Gaussian distribution; and solving the dynamic equation describing the state transition and the equation describing the prediction process based on the current velocity value to obtain the probability distribution of the state transition.

[0137] Finally, step S6 is executed to determine the predicted confidence level of the next state based on the probability distribution of state transitions and the confidence level of the current state. In some embodiments, step S6 specifically includes: recursively calculating the confidence level of the current state using a Kalman filter method based on the probability distribution of state transitions to obtain the predicted confidence level of the next state.

[0138] The problem that confidence updates aim to solve can be described by the following equation:

[0139]

[0140] As introduced in the previous chapters, the likelihood probability distribution p(z) observed by the agent i |x i ,M)~N(μ o ,σ o The probability distribution is also Gaussian, and its variance is estimated by least squares after mapping the sequence cost to probability using the SoftMax equation. The observation model can then be described by the following equation:

[0141] z i =x i +δ i δ i ~M(0,σ o (17)

[0142] Here, the observed state is also a Gaussian linear system, since this scheme defines the observed result as... This result includes Gaussian noise, therefore, the mean of the fitted observations cannot be used as the mean of the probability distribution; this would be meaningless. Here, the variance of the fitted data is a crucial indicator because it reflects the reliability of the observation. If the observation is highly reliable, its probability distribution should be highly concentrated, resulting in a smaller variance. Conversely, if the observation is unreliable, its probability distribution will be more dispersed, leading to a larger variance. Therefore, the variance obtained from the fitted data largely describes the reliability of the match. However, relying solely on the observed results is insufficient. This scheme cannot only consider the situation near the best observation result but also must consider abrupt changes in the observation. If the current observation differs significantly from the previous observation or the predicted result, then the observation is considered unreliable. Therefore, based on the above discussion, this scheme designs the variance of the observations as follows:

[0143]

[0144] Here, the squared difference between the predicted and observed results is used as a penalty factor to evaluate the reliability of the observation, thus preventing observational jumps from affecting the final recognition result. Since this scheme assumes that the likelihood probability distribution of the observed data is also a Gaussian distribution, the updated confidence parameter μ can be calculated using Kalman filtering. i and σ i :

[0145]

[0146] After determining the confidence level of the current state in step S4, the following steps are also performed: Based on the confidence level of the current state, the recognition result of the currently observed image is determined. Therefore, the final image I of the agent is... i The recognition result is I μi ∈M.

[0147] In summary, the key points of this invention are: 1. A method for obtaining a cost matrix based on normalized deep learning descriptors of continuous sequence images and database images; 2. A method for defining sequence cost matching; 3. A method for estimating robot speed based on cosine distance; 4. A method for probabilizing image feature matching similarity; 5. A Markov prediction and update process that simulates human memory and reasoning based on probabilistic statistical features; 6. A design of a prediction variance penalty term to address potential instability and other unexpected situations in the descriptors.

[0148] Embodiments of this application also provide a deep learning-based sequence image scene recognition device, the device comprising:

[0149] The acquisition module acquires image sequences of the currently observed image and the map.

[0150] The extraction module is used to extract image descriptors;

[0151] The cost value calculation module is used to calculate the sequential cost value between the currently observed image and the map based on the extracted descriptors;

[0152] The estimation module is used to estimate the current velocity value based on the sequence cost.

[0153] The first probability determination module is used to determine the current likelihood probability distribution;

[0154] The update module is used to determine the confidence level of the current state based on the current likelihood probability distribution and the prediction confidence level of the previous state.

[0155] The second probability determination module is used to determine the probability distribution of state transitions based on the current velocity value;

[0156] The prediction module is used to determine the prediction confidence of the next state based on the probability distribution of state transitions and the confidence of the current state.

[0157] In some embodiments, the extraction module is specifically used to: process the image using a convolutional neural network model, and use the output of a specified layer as a descriptor; wherein the convolutional neural network model is a model trained on a scene classification dataset; and normalize the descriptor.

[0158] In some embodiments, the cost calculation module is specifically used for:

[0159] Let the set of currently observed images observed by the agent be denoted as: Q = [I1 I2 ... I j ... I n The image sequence of a map is denoted as: M = [I1 I2 ... I i ... I m ];

[0160] Image I i With Image I j The sequence cost value between images I is: i Descriptors and Image I j The value of the cosine distance between the descriptors; where I i ∈M,I j ∈Q.

[0161] In some embodiments, the estimation module is specifically used to: determine a search sequence of a preset length based on the current time; sum the sequence values ​​of the entire search sequence as the degree of matching between the currently observed image and the map; and select the search sequence z with the highest degree of matching by adjusting the slope v of the search sequence. T ; Search sequence z T The corresponding slope v is determined as the agent's current velocity value.

[0162] In some embodiments, the first probability determination module is specifically configured to: search sequence z T Based on the baseline, select N candidate sequences in the vicinity; where N is a preset value; calculate the probability distribution of the N candidate sequences; assuming that the obtained probability distribution is a sample of the Gaussian distribution, estimate the parameters of the Gaussian distribution based on the obtained probability distribution, and use the estimated Gaussian distribution as the current likelihood probability distribution.

[0163] In some embodiments, the update module is specifically used to: recursively calculate the prediction confidence of the previous state based on the current likelihood probability distribution using a Kalman filter method to obtain the confidence of the current state.

[0164] In some embodiments, the scene recognition device further includes a determination module, specifically configured to: after determining the confidence level of the current state, determine the recognition result of the currently observed image based on the confidence level of the current state.

[0165] In some embodiments, the second probability determination module is specifically used to: assume that the probability distribution of the state transition is a Gaussian distribution; and based on the current velocity value, solve the dynamic equation describing the state transition and the equation of the prediction process to obtain the probability distribution of the state transition.

[0166] In some embodiments, the prediction module is specifically used to: recursively calculate the confidence of the current state based on the probability distribution of state transitions using a Kalman filter method to obtain the predicted confidence of the next state.

[0167] Regarding the apparatus in the above embodiments, the specific steps for each module to perform operations have been described in detail in the embodiments related to the method, and will not be elaborated further here. Each module in the above scene recognition apparatus can be implemented entirely or partially through software, hardware, or a combination thereof. Each module can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0168] It is understood that the same or similar parts in the above embodiments can be referred to each other, and the contents not described in detail in some embodiments can be referred to the same or similar contents in other embodiments.

[0169] It should be noted that in the description of this application, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means at least two.

[0170] Any process or method described in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the function involved, as will be understood by those skilled in the art to which embodiments of this application pertain.

[0171] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0172] Those skilled in the art will understand that all or part of the steps of the methods in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, the program includes one or a combination of the steps of the method embodiments.

[0173] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.

[0174] The storage media mentioned above can be read-only memory, disk, or optical disk, etc.

[0175] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0176] Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of this application.

Claims

1. A deep learning-based method for scene recognition of sequential images, characterized in that, include: Obtain the image sequence of the currently observed image and map, and extract the image descriptors; Calculate the sequence cost between the currently observed image and the map based on the extracted descriptors; Estimate the current velocity value based on the sequence cost and determine the current likelihood probability distribution; Determine the confidence level of the current state based on the current likelihood probability distribution and the prediction confidence level of the previous state. Determine the probability distribution of state transitions based on the current velocity value; Based on the probability distribution of state transitions and the confidence of the current state, determine the prediction confidence of the next state; The extracted image descriptor includes: Images are processed using a convolutional neural network model, with the output of a specified layer used as a descriptor; wherein the convolutional neural network model is a model trained on a scene classification dataset; Normalize the descriptors; Estimate the current velocity value based on the sequence cost value, including: Based on the current time, determine a search sequence of a preset length; The summation of sequence values ​​for the entire search sequence is used as the degree of matching between the currently observed image and the map. By the slope of the search sequence v Adjustments were made, and the search sequence with the highest matching degree was selected. ; Search sequence Corresponding slope v This is determined to be the agent's current velocity value; Determining the current likelihood probability distribution includes: With search sequence Based on the benchmark, select the nearby N There are candidate sequences; among which N This is the default value; calculate N The probability distribution of candidate sequences; Assuming the obtained probability distribution is a sample of a Gaussian distribution, the parameters of the Gaussian distribution are estimated based on the obtained probability distribution, and the estimated Gaussian distribution is used as the current likelihood probability distribution.

2. The method according to claim 1, characterized in that, Calculate the sequential cost value between the currently observed image and the map, including: Let the set of currently observed images observed by the agent be denoted as ： The image sequence of the map is denoted as ： ; image With images The sequence cost value between images is: Descriptors and images The value of the cosine distance between the descriptors; where, .

3. The method according to claim 1, characterized in that, Based on the current likelihood probability distribution and the prediction confidence of the previous state, the confidence of the current state is determined, specifically including: Based on the current likelihood probability distribution, the prediction confidence of the previous state is recursively calculated using the Kalman filter method to obtain the confidence of the current state.

4. The method according to claim 3, characterized in that, After determining the confidence level of the current state, the following steps are also included: Based on the confidence level of the current state, determine the recognition result of the currently observed image.

5. The method according to claim 1, characterized in that, The probability distribution of state transitions is determined based on the current velocity value, including: Assume the probability distribution of state transitions follows a Gaussian distribution; Based on the current velocity value, the dynamic equations describing the state transition and the equations predicting the process are solved to obtain the probability distribution of the state transition.

6. The method according to claim 5, characterized in that, Based on the probability distribution of state transitions and the confidence level of the current state, the prediction confidence level of the next state is determined, including: Based on the probability distribution of state transitions, the confidence of the current state is recursively calculated using Kalman filtering to obtain the predicted confidence of the next state.

7. A deep learning-based sequence image scene recognition device, applied to the deep learning-based sequence image scene recognition method according to any one of claims 1-6, characterized in that, include: The acquisition module acquires image sequences of the currently observed image and the map. The extraction module is used to extract image descriptors; The cost value calculation module is used to calculate the sequential cost value between the currently observed image and the map based on the extracted descriptors; The estimation module is used to estimate the current velocity value based on the sequence cost. The first probability determination module is used to determine the current likelihood probability distribution; The update module is used to determine the confidence level of the current state based on the current likelihood probability distribution and the prediction confidence level of the previous state. The second probability determination module is used to determine the probability distribution of state transitions based on the current velocity value; The prediction module is used to determine the prediction confidence of the next state based on the probability distribution of state transitions and the confidence of the current state.

Images