A method and device for constructing a cognitive map

Abstract

The disclosed embodiments provide a method and device for constructing a cognitive map, which relates to the field of artificial intelligence technology. Obtain a stream of images to be processed of a target scene within a historical time period; process the stream of images to be processed based on a visual mileage calculation method to obtain the linear velocity and angular velocity of the robot at multiple historical moments; input the linear velocity at each historical moment into a grid cell annular attractor model to obtain a position feature that characterizes the position of the robot at each historical moment, and input the angular velocity at each historical moment into a head orientation cell model to obtain a posture feature that characterizes the posture of the robot at each historical moment; extract features from the stream of images to be processed based on a Vision Transformer network model to obtain the first image feature of the image frame corresponding to each historical moment in the stream of images to be processed; generate a cognitive map of the target scene based on the obtained position feature, posture feature, and first image feature to improve the accuracy of the cognitive map.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, and in particular to a cognitive map construction method and device. Background Art

[0002] With the rapid development of artificial intelligence technology, it is possible to construct a cognitive map of a scene and guide the robot to travel in the scene based on the map. The simultaneous localization and mapping (SLAM) algorithm is a commonly used map construction algorithm for existing robots. In the process of building a cognitive map, the robot's own posture estimation is very important. For example, in a smart home scenario, after a sweeping robot enters an unfamiliar environment, it can move to every corner of the room without obstacles based on the constructed cognitive map and its own posture, thereby achieving indoor cleaning.

[0003] In related technologies, the current position information of the robot can be obtained through sensor devices such as the Global Positioning System (GPS), Inertial Measurement Unit (IMU), wheel speed sensor and sonar positioning system, and thus a cognitive map of the robot in the current perspective can be constructed.

[0004] However, taking the wheel speed sensor as an example, it is very susceptible to interference from electromagnetic waves, resulting in inaccurate posture information, and thus, the accuracy of the constructed cognitive map is not high. In addition, the current traditional visual SLAM algorithms, such as the 3D positioning and mapping algorithm based on ORB features (ORB, Oriented FAST and Rotated BRIEF), use the bag-of-words model and graph optimization method for closed-loop detection and correction, which has a slow detection speed and poor map accuracy after correction. Summary of the invention

[0005] The purpose of the embodiments of the present disclosure is to provide a method and device for constructing a cognitive map, so as to improve the accuracy of the constructed cognitive map. The specific technical solution is as follows:

[0006] In a first aspect of the embodiments of the present disclosure, a method for constructing a cognitive map is first provided. The method is applied to a robot and comprises:

[0007] During the driving process of the robot, an image stream of a target scene within a historical time period is obtained as an image stream to be processed;

[0008] Based on a visual odometry (VO) algorithm, the image stream to be processed is processed to obtain the linear velocity and angular velocity of the robot at multiple historical moments in the historical time period;

[0009] Inputting the linear velocity at each historical moment into the grid cell annular attractor model to obtain a position feature characterizing the position of the robot at each historical moment, and inputting the angular velocity at each historical moment into the head direction cell model to obtain a posture feature characterizing the posture of the robot at each historical moment;

[0010] Performing feature extraction on the image stream to be processed based on a computer vision processing Vision Transformer network model to obtain image features of an image frame corresponding to each historical moment in the image stream to be processed as a first image feature;

[0011] Based on the obtained position features, posture features and first image features, a cognitive map of the target scene is generated.

[0012] In some embodiments, generating a cognitive map of the target scene based on the obtained position feature, posture feature, and first image feature includes:

[0013] Generate multiple cognitive nodes; wherein one cognitive node includes a position feature, a posture feature, and a first image feature of a corresponding historical moment;

[0014] The multiple cognitive nodes are updated according to a closed-loop update algorithm to obtain a cognitive map of the target scene.

[0015] In some embodiments, during the driving process of the robot, acquiring an image stream of a target scene within a historical time period as an image stream to be processed includes:

[0016] During the driving process of the robot, an image stream of the target scene under the shooting angle of the robot is obtained as the image stream to be processed.

[0017] In some embodiments, during the driving process of the robot, acquiring an image stream of a target scene within a historical time period as an image stream to be processed includes:

[0018] During the driving process of the robot, an image stream of the target scene under the shooting angle of the robot is obtained as an initial image stream;

[0019] For each initial image frame in the initial image stream, performing semantic segmentation on the initial image frame to obtain a first semantic segmentation image;

[0020] Performing feature extraction on the first semantic segmentation image to obtain second image features;

[0021] Based on the first semantic segmentation image and the depth image corresponding to the initial image frame, a corresponding point cloud image is obtained, and features are extracted from the point cloud image to obtain third image features;

[0022] Based on the designated position information, the third image feature and the second image feature are subjected to feature fusion to obtain a second semantic segmentation image; wherein the designated position information represents: a deflection angle and a distance between a designated spatial position and a position when the robot collects the initial image frame;

[0023] Performing feature fusion on the second semantic segmentation image corresponding to the initial image frame and the initial image frame to obtain a fourth image feature;

[0024] Inputting the fourth image feature into an image generation network to obtain a predicted image frame with the specified spatial position as a monitoring perspective;

[0025] The image stream containing the predicted image frames corresponding to each initial image frame is used as the image stream to be processed.

[0026] In some embodiments, the feature extraction of the image stream to be processed based on the computer vision processing Vision Transformer network model to obtain the image features of the image frame corresponding to each historical moment in the image stream to be processed as the first image features includes:

[0027] For each image frame corresponding to a historical moment in the image stream to be processed, dividing the image frame into a plurality of image blocks;

[0028] The multiple image blocks, the position code of each image block, and the category code of the object contained in the image frame are input into the Vision Transformer network model to obtain the first image feature of the image frame.

[0029] A second aspect of the embodiments of the present disclosure provides a cognitive map construction device, which is applied to a robot and includes:

[0030] An image stream acquisition module is used to acquire an image stream of a target scene within a historical time period during the driving process of the robot as an image stream to be processed;

[0031] A visual odometry module processes the image stream to be processed based on a visual odometry calculation method to obtain the linear velocity and angular velocity of the robot at multiple historical moments in the historical time period;

[0032] A cognitive module, for inputting the linear velocity at each historical moment into a grid cell annular attractor model to obtain a position feature characterizing the position of the robot at each historical moment, and for inputting the angular velocity at each historical moment into a head orientation cell model to obtain a posture feature characterizing the posture of the robot at each historical moment;

[0033] A visual feature characterization module extracts features from the image stream to be processed based on a computer vision processing Vision Transformer network model, and obtains image features of an image frame corresponding to each historical moment in the image stream to be processed as a first image feature;

[0034] The scene mapping module is used to generate a cognitive map of the target scene based on the obtained position features, posture features and first image features.

[0035] In some embodiments, the scene mapping module includes:

[0036] A cognitive node generation submodule, used to generate a plurality of cognitive nodes; wherein a cognitive node includes a position feature, a posture feature and a first image feature of a corresponding historical moment;

[0037] The cognitive map updating submodule is used to update the multiple cognitive nodes according to a closed-loop updating algorithm to obtain a cognitive map of the target scene.

[0038] In some embodiments, the image stream acquisition module is specifically used to acquire an image stream of a target scene from a shooting perspective of the robot as an image stream to be processed during the driving process of the robot.

[0039] In some embodiments, the image stream acquisition module is specifically used to:

[0040] During the driving process of the robot, an image stream of the target scene under the shooting angle of the robot is obtained as an initial image stream;

[0041] For each initial image frame in the initial image stream, performing semantic segmentation on the initial image frame to obtain a first semantic segmentation image;

[0042] Performing feature extraction on the first semantic segmentation image to obtain second image features;

[0043] Based on the first semantic segmentation image and the depth image corresponding to the initial image frame, a corresponding point cloud image is obtained, and features are extracted from the point cloud image to obtain third image features;

[0044] Based on the designated position information, the third image feature and the second image feature are subjected to feature fusion to obtain a second semantic segmentation image; wherein the designated position information represents: a deflection angle and a distance between a designated spatial position and a position when the robot collects the initial image frame;

[0045] Performing feature fusion on the second semantic segmentation image corresponding to the initial image frame and the initial image frame to obtain a fourth image feature;

[0046] Inputting the fourth image feature into an image generation network to obtain a predicted image frame with the specified spatial position as a monitoring perspective;

[0047] The image stream containing the predicted image frames corresponding to each initial image frame is used as the image stream to be processed.

[0048] In some embodiments, the visual feature characterization module includes:

[0049] An image block generation submodule, for dividing an image frame corresponding to each historical moment in the image stream to be processed into a plurality of image blocks;

[0050] The first image feature generation submodule is used to input the multiple image blocks, the position code of each image block, and the category code of the object contained in the image frame into the Vision Transformer network model to obtain the first image feature of the image frame.

[0051] In a third aspect of the embodiments of the present disclosure, an electronic device is provided, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0052] Memory, used to store computer programs;

[0053] The processor is used to implement any of the above-mentioned cognitive map construction methods when executing the program stored in the memory.

[0054] In a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any of the above-mentioned cognitive map construction methods is implemented.

[0055] The embodiment of the present disclosure also provides a computer program product including instructions, which, when executed on a computer, enables the computer to execute any of the cognitive map construction methods described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present disclosure, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0057] Figure 1 A schematic diagram of a process of constructing a cognitive map provided by an embodiment of the present disclosure;

[0058] Figure 2 A schematic diagram of a principle of a cognitive map construction method provided in an embodiment of the present disclosure;

[0059] Figure 3 Another flowchart of the cognitive map construction method provided in the embodiment of the present disclosure;

[0060] Figure 4 A schematic diagram of the principle of generating a cognitive map based on an initial image stream provided by an embodiment of the present disclosure;

[0061] Figure 5 A schematic diagram of another process of the cognitive map construction method provided in the embodiment of the present disclosure;

[0062] Figure 6 A structural diagram of a cognitive map construction device provided in an embodiment of the present disclosure;

[0063] Figure 7 A structural diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0064] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field based on the present disclosure belong to the scope of protection of the present disclosure.

[0065] In order to improve the accuracy of cognitive maps, the embodiments of the present disclosure provide a cognitive map construction method and device, which are described in detail below.

[0066] The embodiment of the present disclosure provides a cognitive map construction method, which is applied to a movable robot. The movement process of the robot can also be considered as the driving process of the robot. Figure 1 , Figure 1 A flowchart of a cognitive map construction method provided in an embodiment of the present disclosure may include the following steps:

[0067] S101: During the driving process of the robot, an image stream of a target scene within a historical time period is obtained as an image stream to be processed.

[0068] S102: Based on the visual odometry method, the image stream to be processed is processed to obtain the linear velocity and angular velocity of the robot at multiple historical moments in the historical time period.

[0069] S103: Input the linear velocity at each historical moment into the grid cell annular attractor model to obtain the position feature characterizing the robot's position at each historical moment, and input the angular velocity at each historical moment into the head direction cell model to obtain the posture feature characterizing the robot's posture at each historical moment.

[0070] S104: extracting features from the image stream to be processed based on the computer vision processing Vision Transformer network model, and obtaining image features of the image frame corresponding to each historical moment in the image stream to be processed as the first image features.

[0071] S105: Generate a cognitive map of the target scene based on the obtained position feature, posture feature and first image feature.

[0072] The cognitive map construction method provided by the embodiment of the present disclosure is that the robot can obtain the position characteristics and posture characteristics of the robot through the image stream of the target scene, without the need to measure the posture information of the robot through other sensor devices such as wheel speed sensors, thereby avoiding the problem of low accuracy of posture information caused by the instability of the sensor device. For example, the instability problem caused by the interference of electromagnetic waves on the sensor device in the related technology can be avoided. In addition, it can also avoid the problem of low accuracy of posture information caused by the influence of light on the sensor device and the drift of the sensor device. Thus, the accuracy of the cognitive map is improved. Furthermore, the embodiment of the present application no longer uses the visual SLAM algorithm, which avoids the problem of slow detection speed and poor accuracy of the corrected map caused by using the bag of words model and graph optimization method for closed-loop detection and correction.

[0073] The method provided by the embodiments of the present disclosure can be combined with 5G and 6G technologies to achieve tasks such as autonomous environmental modeling, intelligent recognition, situational memory, object-oriented navigation, etc. in dynamic and complex environments at a faster speed.

[0074] For step S101, the target scene is the scene for which a cognitive map needs to be generated for the current unfamiliar environment. The target scene can be an indoor scene or an outdoor scene. For example, the target scene can be a user's residence, and the corresponding robot can be a sweeping robot; the target scene can also be an unmanned dock, and the corresponding robot can be a robot for transporting goods. The image stream to be processed usually contains multiple image frames.

[0075] For step S102, based on the visual odometry method, for one image frame, by comparing the image frame with the previous image frame, the linear velocity and angular velocity of the robot at the corresponding moment of the image frame are calculated.

[0076] In one implementation, the robot can obtain an image frame A and an image frame B preceding image frame A from the image stream to be processed. Then, the robot can perform distortion calibration on image frame A and image frame B respectively based on the image distortion algorithm. Then, the image features of the calibrated image frame A and image frame B can be extracted respectively by using the image feature extraction algorithm, and the extracted image features can be matched, and an optical flow field can be constructed based on the matching results. Then, the motion of the robot is estimated by the optical flow field, so as to obtain the linear velocity and angular velocity corresponding to image frame A.

[0077] For example, the image distortion algorithm may be an edge straight line fitting algorithm; and the image feature extraction algorithm may be a fast feature detection (FAST, Features from Accelerated Segment Test) algorithm.

[0078] With respect to step S103, for example, the grid cell annular attractor model may be a continuous attractor neural network model (CANN, Continuous Attractor Neural Network), which is a regularized neural computing model.

[0079] In the related art, the position feature can be obtained based on the linear velocity of the robot through the position cell model. The grid cell annular attractor model can include multiple grid cells (Grid cell), and the discharge positions of the grid cells can constitute a hexagonal grid coordinate system. There is a certain distance between adjacent discharge positions, and this distance is called the grid spacing of the grid cells. Compared with the position cell model, the grid cell annular attractor model can not only encode the position information of the robot, but also the grid in the grid cell annular attractor model will change with the familiarity with the space. Based on the input linear velocity, the grid cell annular attractor model can obtain a dynamic and continuous position expression, that is, the grid cell annular attractor model can establish a coordinate system in conjunction with the input information to determine the relative position of the object.

[0080] In one implementation, for each historical moment, the linear velocity at that historical moment can be input into the grid cell annular attractor model to obtain the discharge activity of the grid cell. The discharge activity includes the size of the grid cell and the grid spacing of the grid cell. Then, based on the size of the grid cell and the grid spacing, the robot's current self-position perception is formed, that is, the current position feature representing the robot is calculated. For example, the position feature can be represented by a matrix containing the position information.

[0081] In one implementation, a head-to-head cell model may be generated based on a continuous attractor algorithm. The activity of the head-to-head cell model represents the movement angle of the robot. The activity of the head-to-head cell model may be represented by a one-dimensional vector (which may be referred to as an excitation vector of the head-to-head cell model). In addition, the activity of the head-to-head cell model may be updated according to continuous attraction dynamics. For example, in the head-to-head cell model, the excitation vector of the head-to-head cell model may be represented by formula (1):

[0082]

[0083] Where a represents the index in the coordinate system of the head facing the cell model; k hd Indicates the width constant of the head-to-cell model. For example, the width constant may be 7; ε hd represents the excitation vector of the head-facing cell model, and e is a constant.

[0084] The head-facing cell model excitation vector can be updated based on formula (2):

[0085]

[0086] Among them, n θ′ Indicates the dimension of the head-facing cell model, for example, it can be 36; H k represents the activity of the head-facing cell model when the index of the coordinate system is k; ΔH θ， Represents the update amount of the head heading cell model under excitatory conditions.

[0087] With respect to step S104, the Vision Transformer network model is a type of computer vision feature extraction network. For example, the Vision Transformer network model used in the embodiments of the present disclosure may be a computer vision feature extraction network such as a Swin transformer network model and a C-Tran network model.

[0088] See also Figure 2 , Figure 2 A schematic diagram of the principle of the cognitive map construction method provided in the embodiment of the present disclosure.

[0089] Figure 2In the example, the image stream of the target scene in the historical time period (i.e., the image stream to be processed) can be obtained, and the image stream can be processed based on the visual odometer module to obtain the angular velocity and linear velocity at multiple historical moments. Then, the angular velocities at multiple historical moments can be input into the head heading cell module, i.e. Figure 1 In the embodiment shown, the head is directed to the cell model, the posture feature is obtained, and the linear velocity at multiple historical moments is input into the grid cell module. Figure 1 In the grid cell annular attractor model in the illustrated embodiment, the position feature is obtained. Furthermore, the image stream is input into the C-Tran network for feature extraction to obtain the first image feature. Furthermore, the robot generates a cognitive map based on the position feature, the posture feature and the first image feature.

[0090] In some embodiments, see Figure 3 , Figure 3 Another flowchart of the cognitive map construction method provided in the embodiment of the present disclosure is shown in FIG. Figure 1 On the basis of, the above step S105 may include the following steps:

[0091] S1051: Generate multiple cognitive nodes.

[0092] Among them, a cognitive node includes the position feature, posture feature and first image feature of the corresponding historical moment.

[0093] S1052: Update multiple cognitive nodes according to a closed-loop update algorithm to obtain a cognitive map of the target scene.

[0094] In the disclosed embodiment, the cognitive map is composed of a plurality of cognitive nodes, each of which contains a position feature, a posture feature and a first image feature at a moment. The construction of the cognitive map is actually a process of continuously updating cognitive nodes.

[0095] For ease of description, in the embodiments of the present disclosure, the position features, posture features, and first image features of a historical moment may be collectively referred to as cognitive feature data of the historical moment.

[0096] In one implementation, each time cognitive feature data of a historical moment is obtained, a difference judgment can be made between the cognitive feature data and the cognitive nodes currently existing in the cognitive map to determine whether the cognitive feature data meets the update condition. If the cognitive feature data meets the update condition, the cognitive node corresponding to the cognitive feature data is added to the cognitive map; if the cognitive feature data does not meet the update condition, that is, meets the closed-loop condition, the cognitive nodes currently existing in the cognitive map are updated. For example, a cognitive node (which can be referred to as a cognitive node to be updated) whose corresponding position in the cognitive node currently existing in the cognitive map is consistent with the position at the historical moment can be determined, and the cognitive node to be updated is replaced with the cognitive node corresponding to the cognitive feature data. At this time, no new cognitive node will be added to the cognitive map.

[0097] In one implementation, when each cognitive feature data is obtained, the difference between the cognitive feature data and each cognitive node currently existing in the cognitive map can be compared based on the difference function to obtain the corresponding difference degree; if each difference degree is greater than a threshold, it is determined that the update condition is met. Otherwise, it is determined that the update condition is not met.

[0098] For example, the difference between the cognitive feature data and each cognitive node (which may be referred to as a target cognitive node) currently existing in the cognitive map may be calculated based on formula (3):

[0099] S=μ d |D′-D|+μ a |A′-A|+μ v ·|V′-V| (3)

[0100] S represents the difference, μ d represents the weight of the position feature, D represents the position feature in the cognitive feature data, D′ represents the position feature of the target cognitive node, μ a represents the weight of the posture feature, A represents the posture feature in the cognitive feature data, A′ represents the posture feature of the target cognitive node, μ v represents the weight of the first image feature, V represents the first image feature in the cognitive feature data, and V′ represents the first image feature of the target cognitive node.

[0101] The robot can obtain the position and posture features of the robot through the image stream of the target scene, without the need to measure the robot's posture information through other sensor devices such as wheel speed sensors, thereby avoiding the problem of low accuracy of posture information caused by the instability of sensor devices. For example, it can avoid the instability problem caused by electromagnetic interference of sensor devices in related technologies. In addition, it can also avoid the problem of low accuracy of posture information caused by the influence of light on sensor devices and drift of sensor devices. In turn, the accuracy of cognitive maps is improved.

[0102] In one implementation, for step S101, the image stream to be processed may be an image stream of a target scene under the shooting perspective of the robot. In this case, the generated cognitive map represents the cognitive map under the current shooting perspective of the robot.

[0103] Alternatively, the image stream to be processed may be an image stream of a target scene from a perspective of a specified spatial position. In this case, the generated cognitive map represents a cognitive map from a perspective of the specified spatial position.

[0104] For example, the robot may be provided with an image acquisition device, through which an image stream (i.e., an initial image stream) of a target scene under a shooting angle of view during driving may be acquired. The initial image stream is used as an image stream to be processed, and then, based on the above steps S102-S105, a cognitive map of the target scene may be generated.

[0105] In one implementation, step S101 may include the following steps:

[0106] Step 1: During the robot's driving process, obtain the image stream of the target scene from the robot's shooting perspective as the initial image stream.

[0107] Step 2: For each initial image frame in the initial image stream, semantic segmentation is performed on the initial image frame to obtain a first semantic segmentation image.

[0108] Step 3: Extract features from the first semantic segmentation image to obtain second image features.

[0109] Step 4: Based on the first semantic segmentation image and the depth image corresponding to the initial image frame, a corresponding point cloud image is obtained, and features are extracted from the point cloud image to obtain third image features.

[0110] Step 5: Based on the specified position information, the third image feature and the second image feature are subjected to feature fusion to obtain a second semantic segmentation image; wherein the specified position information represents: a deflection angle and a distance between the specified spatial position and the position when the robot collected the initial image frame.

[0111] Step six: perform feature fusion on the second semantic segmentation image corresponding to the initial image frame and the initial image frame to obtain a fourth image feature.

[0112] Step 7: Input the fourth image feature into the image generation network to obtain a predicted image frame with the specified spatial position as the monitoring perspective.

[0113] Step 8: Using the image stream containing the predicted image frames corresponding to each initial image frame as the image stream to be processed.

[0114] The designated spatial position can be set by the technician according to the requirements. For example, the designated spatial position can be determined based on the current position of the robot and the designated position information. That is, the technician can set the designated position information in advance. For example, starting from the current position, the designated spatial position can be obtained by moving according to the deflection angle and distance.

[0115] Based on semantic segmentation, the image area occupied by the object in the initial image frame and the category of the object can be determined. The object in the image frame can be a table, a chair, and other objects in the scene.

[0116] The depth image corresponding to the initial image frame represents: a depth image of the same object as that contained in the initial image frame. The value of a pixel in the depth image represents the distance between the corresponding position in the target scene and the robot.

[0117] The point cloud image can be regarded as a data set, and each point in the data set contains: a three-dimensional coordinate and the category of the object corresponding to the three-dimensional coordinate.

[0118] Based on the above processing, the predicted image frame of the specified spatial position can be predicted through the initial image frame of the current perspective. Furthermore, based on the above steps S102-S105, a cognitive map with the specified spatial position as the monitoring perspective can be obtained, so that when the robot enters an unfamiliar environment, it can predict the environmental image under an unknown perspective. Furthermore, the robot does not need to move to the specified spatial position to obtain a cognitive map with the specified spatial position as the monitoring perspective, which can reduce the robot's workload of exploring the environment, help the robot adapt to unfamiliar environments faster, and improve the efficiency of building cognitive maps.

[0119] In one implementation, after obtaining the first semantic segmentation image, feature extraction may be performed on the first semantic segmentation image based on a convolutional layer and an encoder to obtain second image features.

[0120] After obtaining the point cloud image, feature extraction can be performed on the point cloud image based on the encoder and the fully connected layer to obtain the third image feature.

[0121] Furthermore, based on the designated position information, the second image features and the third image features may be fused, and the fusion result may be input into the decoder to obtain a second semantic segmentation image.

[0122] In addition, feature extraction can be performed on the initial image frame and the second semantic segmentation image based on the convolution layer, and feature fusion can be performed on the feature extraction results to obtain a fourth image feature.

[0123] The encoder and decoder mentioned in the above process can be implemented by a random encoder-decoder network. The image generation network can be a generative adversarial network (GAN).

[0124] For example, see Figure 4 , Figure 4 A schematic diagram of the principle of generating a cognitive map based on an initial image stream provided in an embodiment of the present disclosure.

[0125] Figure 4 In the process, the robot obtains the initial image stream and the designated position information contained in the initial image stream, and performs semantic segmentation on the initial image frame in the initial image stream to obtain the corresponding semantic segmentation image (i.e., the first semantic segmentation image). The first semantic segmentation image is input into the convolution layer 1 and the encoder 1 for feature extraction to obtain the image features of the first semantic segmentation image (i.e., the second image features); in addition, the corresponding point cloud image is obtained based on the first semantic segmentation image and the depth image corresponding to the initial image frame, and the point cloud image is input into the encoder 2 and the fully connected layer to perform feature extraction on the point cloud image to obtain the image features corresponding to the point cloud image (i.e., the third image features). The second image features and the third image features can be fused, and the fusion results are input into the decoder 3 to generate a predicted semantic segmentation image (i.e., the second semantic segmentation image). The predicted semantic segmentation image and the initial image frame are input into the convolution layer 2 for feature extraction respectively, and the feature extraction results are fused to obtain the fourth image features. The fourth image features are input into the image generation network to generate an RGB image with the specified spatial position as the monitoring perspective, that is, a predicted image frame with the specified spatial position as the monitoring perspective. Each predicted image frame is input into the scene construction module to obtain a cognitive map of the target scene under the monitoring perspective with the designated spatial position. The scene construction module can process based on the above steps S102-S105 to obtain a corresponding cognitive map.

[0126] In one implementation, see Figure 5 , Figure 5 Another flowchart of the cognitive map construction method provided in the embodiment of the present disclosure is shown in FIG. Figure 1 On the basis of, the above step S104 may include the following steps:

[0127] S1041 , for an image frame corresponding to each historical moment in the image stream to be processed, divide the image frame into a plurality of image blocks.

[0128] S1042, input multiple image blocks, the position code of each image block, and the category code of the object contained in the image frame into the Vision Transformer network model to obtain the first image feature of the image frame.

[0129] The position code of each image block is used to indicate the position of the image block in the image frame, and the category code of the object is used to indicate the category of the object.

[0130] In one implementation, the dimensions of an image frame can be represented by H×W×C, where H represents the height of the image frame, W represents the width of the image frame, and C represents the number of channels. For each image frame, the image frame can be divided into N image patches. Each image patch can have a corresponding dimension of P 2 Then, the first eigenvectors corresponding to the N image blocks are concatenated to obtain a vector with a dimension of N×(P 2 Then, the N image blocks, the position code of each image block, and the category code of the object contained in the image frame are input into the VisionTransformer network model.

[0131] The Vision Transformer network model may include an Encoder module, which may include multiple layers, each of which may include a multi-head self-attention module (MSA) and a multi-layer perceptron module (MLP). In addition, each layer in the Encoder module can obtain the output result of the previous layer, perform a normalization operation, input the normalized result into the multi-head self-attention module, and calculate the output matrix. Then, the output matrix is ​​integrated with the output result of the previous layer, the integrated result is layer normalized, and the normalized result is input into the multi-layer perceptron module to obtain the output result of the layer as the input data of the next layer.

[0132] It can be understood that the input data of the first layer includes the above-mentioned multiple image blocks, the position code of each image block, and the category code of the object contained in the image frame. The output result of the last layer is used as the first image feature.

[0133] In one implementation, the first feature vector can be transformed based on Linear Projection to obtain a second feature vector of fixed length D, so as to avoid the Vision Transformer network model being affected by the patch size. That is, the H×W×C image frame is transformed into N D-dimensional second feature vectors.

[0134] For example, the first image feature can be obtained by the following formula:

[0135]

[0136]

[0137] Z′ l =MSA(LN(Z l-1 ))+Z l-1 l=1…L

[0138] Z l =MLP(LN(Z′1)) l=1…L

[0139] y=LN(Z L )

[0140] Among them, Z0 represents the output result of the first layer in the Encoder module, which represents an image block, and X cls Represents the category code of the object contained in the image frame, E represents the transformation matrix of Linear Projection, The encoding of the position information between the Nth image block and the Nth image block, E pos represents a placeholder, D represents the dimension of the second eigenvector, and P represents the side length of each image block. l-1 Represents the output result of the lth layer in the Encoder module, Z′ l represents the integration result of the l+1th layer, Z l Represents the output result of the l+1th layer, MSA represents the multi-head attention module, MLP represents the multi-layer perceptron module, LN represents the normalization operation, L+1 represents the number of layers contained in the Encoder module, and y represents the first image feature.

[0141] Wherein, N can be obtained by formula (4).

[0142]

[0143] Based on the same inventive concept, the present disclosure also provides a cognitive map construction device, see Figure 6 , Figure 6 A structural diagram of a cognitive map construction device provided in an embodiment of the present disclosure, the device comprising:

[0144] The image stream acquisition module 601 is used to acquire the image stream of the target scene within the historical time period during the robot's driving process as the image stream to be processed;

[0145] The visual odometry module 602 is used to process the image stream to be processed based on the visual odometry calculation method to obtain the linear velocity and angular velocity of the robot at multiple historical moments in the historical time period;

[0146] The cognitive module 603 is used to input the linear velocity at each historical moment into the grid cell annular attractor model to obtain the position feature representing the robot position at each historical moment, and input the angular velocity at each historical moment into the head direction cell model to obtain the posture feature representing the robot posture at each historical moment;

[0147] A visual feature characterization module 604 is used to extract features from the image stream to be processed based on a computer vision processing Vision Transformer network model, and obtain image features of an image frame corresponding to each historical moment in the image stream to be processed as a first image feature;

[0148] The scene mapping module 605 is used to generate a cognitive map of the target scene based on the obtained position features, posture features and first image features.

[0149] In some embodiments, the scene mapping module 605 includes:

[0150] A cognitive node generation submodule, used to generate a plurality of cognitive nodes; wherein a cognitive node includes a position feature, a posture feature and a first image feature of a corresponding historical moment;

[0151] The cognitive map update submodule is used to update multiple cognitive nodes according to a closed-loop update algorithm to obtain a cognitive map of the target scene.

[0152] In some embodiments, the image stream acquisition module 601 is specifically used to acquire an image stream of a target scene from the shooting perspective of the robot during the driving process of the robot as an image stream to be processed.

[0153] In some embodiments, the image stream acquisition module 601 is specifically used to:

[0154] During the robot's driving process, an image stream of the target scene from the robot's shooting angle is obtained as an initial image stream;

[0155] For each initial image frame in the initial image stream, semantically segment the initial image frame to obtain a first semantically segmented image;

[0156] Performing feature extraction on the first semantic segmentation image to obtain second image features;

[0157] Based on the first semantic segmentation image and the depth image corresponding to the initial image frame, a corresponding point cloud image is obtained, and features are extracted from the point cloud image to obtain third image features;

[0158] Based on the designated position information, feature fusion is performed on the third image feature and the second image feature to obtain a second semantic segmentation image; wherein the designated position information represents: a deflection angle and a distance between the designated spatial position and the position when the robot collects the initial image frame;

[0159] Performing feature fusion on the second semantic segmentation image corresponding to the initial image frame and the initial image frame to obtain a fourth image feature;

[0160] Inputting the fourth image feature into an image generation network to obtain a predicted image frame with the specified spatial position as a monitoring perspective;

[0161] The image stream containing the predicted image frames corresponding to each initial image frame is used as the image stream to be processed.

[0162] In some embodiments, the visual feature characterization module 604 includes:

[0163] An image block generation submodule, for dividing an image frame corresponding to each historical moment in the image stream to be processed into a plurality of image blocks;

[0164] The first image feature generation submodule is used to input multiple image blocks, the position code of each image block, and the category code of the object contained in the image frame into the Vision Transformer network model to obtain the first image feature of the image frame.

[0165] The present disclosure also provides an electronic device, such as Figure 7 As shown, it includes a processor 701, a communication interface 702, a memory 703 and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704.

[0166] Memory 703, used for storing computer programs;

[0167] The processor 701 is configured to implement the steps of any cognitive map construction method in the above-mentioned embodiments when executing the program stored in the memory 703 .

[0168] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0169] The communication interface is used for communication between the above electronic device and other devices.

[0170] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0171] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0172] In another embodiment provided by the present disclosure, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned cognitive map construction methods are implemented.

[0173] In another embodiment provided by the present disclosure, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any cognitive map construction method in the above embodiments.

[0174] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present disclosure is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0175] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0176] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0177] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the protection scope of the present disclosure. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present disclosure are included in the protection scope of the present disclosure.

Claims

1. A cognitive map construction method, characterized in that: The method is applied to a robot, and the method comprises: During the driving process of the robot, an image stream of the target scene under the shooting angle of view of the robot is obtained as an initial image stream; for each initial image frame in the initial image stream, the initial image frame is semantically segmented to obtain a first semantic segmentation image; feature extraction is performed on the first semantic segmentation image to obtain a second image feature; based on the first semantic segmentation image and the depth image corresponding to the initial image frame, a corresponding point cloud image is obtained, and feature extraction is performed on the point cloud image to obtain a third image feature; based on the specified position information, the third image feature and the second image feature are feature fused to obtain a second semantic segmentation image; wherein the specified position information represents: the deflection angle and distance between the specified spatial position and the position when the robot collects the initial image frame; feature fusion is performed on the second semantic segmentation image corresponding to the initial image frame and the initial image frame to obtain a fourth image feature; the fourth image feature is input into the image generation network to obtain a predicted image frame with the specified spatial position as the monitoring angle of view; the image stream containing the predicted image frames corresponding to each initial image frame is used as the image stream to be processed; Based on the visual odometry calculation method, the image stream to be processed is processed to obtain the linear velocity and angular velocity of the robot at multiple historical moments in a historical time period; Inputting the linear velocity at each historical moment into the grid cell annular attractor model to obtain a position feature characterizing the position of the robot at each historical moment, and inputting the angular velocity at each historical moment into the head orientation cell model to obtain a posture feature characterizing the posture of the robot at each historical moment; Performing feature extraction on the image stream to be processed based on a computer vision processing Vision Transformer network model to obtain image features of an image frame corresponding to each historical moment in the image stream to be processed as a first image feature; Based on the obtained position features, posture features and first image features, a cognitive map of the target scene is generated.

2. The method according to claim 1, characterized in that The step of generating a cognitive map of the target scene based on the obtained position feature, posture feature and first image feature comprises: Generate multiple cognitive nodes; wherein one cognitive node includes a position feature, a posture feature, and a first image feature of a corresponding historical moment; The multiple cognitive nodes are updated according to a closed-loop update algorithm to obtain a cognitive map of the target scene.

3. The method according to claim 1, characterized in that The process of the robot driving is to obtain an image stream of the target scene in a historical time period as an image stream to be processed, including: During the driving process of the robot, an image stream of the target scene under the shooting angle of the robot is obtained as the image stream to be processed.

4. The method according to claim 1, characterized in that: The computer vision processing VisionTransformer network model is used to extract features from the image stream to be processed, and image features of image frames corresponding to each historical moment in the image stream to be processed are obtained as first image features, including: For each image frame corresponding to a historical moment in the image stream to be processed, dividing the image frame into a plurality of image blocks; The multiple image blocks, the position code of each image block, and the category code of the object contained in the image frame are input into the Vision Transformer network model to obtain the first image feature of the image frame.

5. A cognitive map construction device, characterized in that: The device is applied to a robot, and comprises: An image stream acquisition module is used to acquire an image stream of a target scene from the shooting perspective of the robot as an initial image stream during the driving process of the robot; for each initial image frame in the initial image stream, perform semantic segmentation on the initial image frame to obtain a first semantic segmentation image; perform feature extraction on the first semantic segmentation image to obtain a second image feature; based on the first semantic segmentation image and the depth image corresponding to the initial image frame, obtain a corresponding point cloud image, and perform feature extraction on the point cloud image to obtain a third image feature; based on the specified position information, perform feature fusion on the third image feature and the second image feature to obtain a second semantic segmentation image; wherein the specified position information represents: a deflection angle and distance between a specified spatial position and the position of the robot when collecting the initial image frame; perform feature fusion on the second semantic segmentation image corresponding to the initial image frame and the initial image frame to obtain a fourth image feature; input the fourth image feature into an image generation network to obtain a predicted image frame with the specified spatial position as the monitoring perspective; use the image stream containing the predicted image frames corresponding to each initial image frame as the image stream to be processed; A visual odometry module, used to process the image stream to be processed based on a visual odometry calculation method, and obtain the linear velocity and angular velocity of the robot at multiple historical moments in a historical time period; A cognitive module, for inputting the linear velocity at each historical moment into a grid cell annular attractor model to obtain a position feature characterizing the position of the robot at each historical moment, and for inputting the angular velocity at each historical moment into a head orientation cell model to obtain a posture feature characterizing the posture of the robot at each historical moment; A visual feature characterization module, used for extracting features from the image stream to be processed based on a computer vision processing Vision Transformer network model, and obtaining image features of an image frame corresponding to each historical moment in the image stream to be processed as a first image feature; The scene mapping module is used to generate a cognitive map of the target scene based on the obtained position features, posture features and first image features.

6. The device according to claim 5, characterized in that The scene mapping module includes: A cognitive node generation submodule, used to generate a plurality of cognitive nodes; wherein a cognitive node includes a position feature, a posture feature and a first image feature of a corresponding historical moment; The cognitive map updating submodule is used to update the multiple cognitive nodes according to a closed-loop updating algorithm to obtain a cognitive map of the target scene.

7. The device according to claim 5, characterized in that The image stream acquisition module is specifically used to acquire the image stream of the target scene from the shooting angle of the robot during the driving process of the robot as the image stream to be processed.

8. The device according to claim 5, characterized in that The visual feature representation module comprises: An image block generation submodule, for dividing an image frame corresponding to each historical moment in the image stream to be processed into a plurality of image blocks; The first image feature generation submodule is used to input the multiple image blocks, the position code of each image block, and the object category code contained in the image frame into the Vision Transformer network model to obtain the first image feature of the image frame.

9. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 4 when executing a program stored in a memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 4 are implemented.

Images