An open object class perception method and device serving a robot scene

Abstract

The application discloses an open object category perception method and device serving a robot scene. In the method, an image block of an object is obtained through image segmentation and clustering, so that the open object position in the image can be located, finally, the category of the image block is taken as the category of the object corresponding to the image block, so that the open object category can be perceived, and since the obtained open object category is not specific, the non-specific open object category can be detected, and the perception ability of the open object category is improved.

Description

Technical Field

[0001] This invention relates to the field of object recognition technology, and more specifically, to an open object category perception method and apparatus for robot scenarios. Background Technology

[0002] For mobile robots, the ability to perceive the categories of objects is crucial. For example, if a robot is a vehicle, then perceiving the categories of objects on the road while driving helps ensure its safety.

[0003] Existing object recognition methods have the following limitations:

[0004] 1. It can only identify specific, frequently encountered common object categories, such as: vehicles, pedestrians, cyclists, and traffic signs;

[0005] 2. It can identify the object category of specific open objects, but lacks the ability to locate the position of open objects in the image. The open objects are objects that are not common in certain scenarios, such as mineral water bottles in a driving lane.

[0006] 3. It can identify the object category of open objects, but its ability to detect open category objects is limited because the labeled dataset only contains a limited number of categories.

[0007] It is evident that existing object recognition methods have poor ability to perceive open object categories. Summary of the Invention

[0008] This invention provides a method and apparatus for perceiving open object categories in robotic scenarios, which can improve the ability to perceive open object categories. The specific technical solution is as follows.

[0009] In a first aspect, the present invention provides an open object category perception method for robot scenarios, comprising:

[0010] Obtain the point cloud sequence and the corresponding image sequence;

[0011] For each point cloud in the point cloud sequence, each point cloud point is projected onto the corresponding image frame according to a preset data preprocessing method to obtain the point cloud projection point;

[0012] For each image frame in the image sequence, the image frame is segmented according to a preset image segmentation method and all point cloud projection points in the image frame to obtain the image segmentation result of the image frame;

[0013] For each point cloud in the point cloud sequence, each point cloud point in the point cloud is taken as a node of the graph, and the geometric edges and semantic edges contained in the graph are determined based on the image segmentation results of the image frame corresponding to the point cloud.

[0014] Clustering is performed based on a preset clustering method, the geometric edges, and the semantic edges to obtain the clustering results of the objects in the graph;

[0015] For each object in the clustering result, the object is projected onto the image sequence to obtain the corresponding image patch;

[0016] For each image patch, the image patch is input into a preset image classification network to obtain the feature map corresponding to the image patch. The point cloud points of the object corresponding to the image patch are projected onto the image patch to obtain the image patch feature vector. Multiple open object category feature vectors are obtained. The category of the image patch is determined based on the image patch feature vector and the object category feature vector, and the category of the image patch is used as the category of the object corresponding to the image patch.

[0017] Optionally, the step of projecting each point cloud in the point cloud sequence onto a corresponding image frame to obtain the point cloud projection point according to a preset data preprocessing method includes:

[0018] Acquire the intrinsic and extrinsic parameters of the camera that captured the image sequence and the pose information of the lidar that acquired the point cloud sequence;

[0019] For each point cloud in the point cloud sequence, based on the pose information of the lidar when the point cloud was acquired and the intrinsic and extrinsic parameters of the camera, each point cloud point in the point cloud is projected into the corresponding image frame to obtain the point cloud projection point.

[0020] Optionally, the preset image segmentation method is either the arbitrary segmentation model SAM or the Mask2Former image segmentation model.

[0021] Optionally, the step of determining the geometric and semantic edges contained in the graph based on the image segmentation result of the image frame corresponding to the point cloud includes:

[0022] Point cloud point pairs corresponding to point cloud projection point pairs belonging to the same time and the same image segmentation result are taken as a semantic edge. The Euclidean distance between every two point cloud points in the graph is calculated. Point cloud point pairs belonging to different time and whose Euclidean distance is less than a preset distance threshold are taken as a geometric edge.

[0023] Optionally, the preset clustering method is a spectral clustering method.

[0024] Optionally, the step of projecting the object into the image sequence to obtain the image patch corresponding to the object includes:

[0025] Based on the pose information of the lidar when collecting the point cloud corresponding to the object, as well as the intrinsic and extrinsic parameters of the camera, the object is projected into the image sequence, the minimum bounding rectangle region corresponding to the object in each image frame is determined, and the image block corresponding to the object is determined based on the minimum bounding rectangle region.

[0026] Optionally, the preset image classification network is a contrastive language-image pre-trained CLIP network or an open contrastive language-image pre-trained OpenCLIP network.

[0027] Optionally, the Transformer of the text encoder in the contrastive language-image pre-trained CLIP network is calculated as follows:

[0028] Attn VV =softmax(V*V) T )*V

[0029] Among them, Attn VV Let V be the attention weight matrix, where V is the value, T is the transpose, and softmax is the normalization function.

[0030] Optionally, the step of obtaining object category feature vectors for multiple open object categories and determining the category of the image patch based on the image patch feature vector and the object category feature vector includes:

[0031] Based on the category names of multiple open object categories and a preset feature vector extraction algorithm, the object category feature vector for each open object category is obtained;

[0032] Calculate the similarity between each object category feature vector and the image patch feature vector, and take the open object category corresponding to the object category feature vector with the highest similarity as the category of the image patch.

[0033] Secondly, the present invention provides an open object category perception device for robotic scenarios, comprising:

[0034] The acquisition module is used to acquire point cloud sequences and corresponding image sequences;

[0035] The preprocessing module is used to project each point cloud in the point cloud sequence onto the corresponding image frame to obtain the point cloud projection point according to a preset data preprocessing method.

[0036] The image segmentation module is used to segment each image frame in the image sequence according to a preset image segmentation method and all point cloud projection points in the image frame, so as to obtain the image segmentation result of the image frame.

[0037] The graph optimization module is used to take each point cloud in the point cloud sequence as a node of the graph, and determine the geometric edges and semantic edges contained in the graph based on the image segmentation result of the image frame corresponding to the point cloud.

[0038] The clustering module is used to perform clustering based on a preset clustering method, the geometric edges, and the semantic edges to obtain the clustering results of the objects in the graph;

[0039] The projection module is used to project each object in the clustering result onto the image sequence to obtain the image patch corresponding to the object;

[0040] The category-aware module is used to input each image patch into a preset image classification network to obtain the feature map corresponding to the image patch, project the point cloud points of the object corresponding to the image patch onto the image patch to obtain the image patch feature vector, obtain the object category feature vectors of multiple open object categories, determine the category of the image patch based on the image patch feature vector and the object category feature vector, and use the category of the image patch as the category of the object corresponding to the image patch.

[0041] Optionally, the preprocessing module includes:

[0042] The acquisition submodule is used to acquire the intrinsic and extrinsic parameters of the camera that captures the image sequence and the pose information of the lidar that collects the point cloud sequence.

[0043] The projection submodule is used to project each point cloud in the point cloud sequence onto the corresponding image frame to obtain the point cloud projection point, based on the pose information of the lidar when the point cloud was acquired and the intrinsic and extrinsic parameters of the camera.

[0044] Optionally, the preset image segmentation method is either the arbitrary segmentation model SAM or the Mask2Former image segmentation model.

[0045] Optionally, the graph optimization module is specifically used for:

[0046] Point cloud point pairs corresponding to point cloud projection point pairs belonging to the same time and the same image segmentation result are taken as a semantic edge. The Euclidean distance between every two point cloud points in the graph is calculated. Point cloud point pairs belonging to different time and whose Euclidean distance is less than a preset distance threshold are taken as a geometric edge.

[0047] Optionally, the preset clustering method is a spectral clustering method.

[0048] Optionally, the projection module is specifically used for:

[0049] Based on the pose information of the lidar when collecting the point cloud corresponding to the object, as well as the intrinsic and extrinsic parameters of the camera, the object is projected into the image sequence, the minimum bounding rectangle region corresponding to the object in each image frame is determined, and the image block corresponding to the object is determined based on the minimum bounding rectangle region.

[0050] Optionally, the preset image classification network is a contrastive language-image pre-trained CLIP network or an open contrastive language-image pre-trained OpenCLIP network.

[0051] Optionally, the Transformer of the text encoder in the contrastive language-image pre-trained CLIP network is calculated as follows:

[0052] Attn VV =softmax(V*V) T )*V

[0053] Among them, Attn VV Let V be the attention weight matrix, where V is the value, T is the transpose, and softmax is the normalization function.

[0054] Optionally, the category-aware module includes:

[0055] The feature extraction submodule is used to obtain the object category feature vector for each open object category based on the category names of multiple open object categories and a preset feature vector extraction algorithm.

[0056] The calculation submodule is used to calculate the similarity between each object category feature vector and the image patch feature vector, and to take the open object category corresponding to the object category feature vector with the highest similarity as the category of the image patch.

[0057] As described above, the open object category perception method and apparatus for robot scenarios provided by this invention can acquire point cloud sequences and corresponding image sequences. For each point cloud in the point cloud sequence, each point cloud point is projected onto the corresponding image frame according to a preset data preprocessing method to obtain point cloud projection points. For each image frame in the image sequence, the image frame is segmented according to a preset image segmentation method and all point cloud projection points in the image frame to obtain the image segmentation result of the image frame. For each point cloud in the point cloud sequence, each point cloud point is used as a node in the graph, and the image segmentation result of the corresponding image frame is used to determine... The geometric and semantic edges contained in the image are clustered according to a preset clustering method, geometric edges, and semantic edges to obtain the clustering results of the objects in the image. For each object in the clustering results, the object is projected into the image sequence to obtain the corresponding image patch. For each image patch, the image patch is input into a preset image classification network to obtain the corresponding feature map. The point cloud points of the object corresponding to the image patch are projected onto the image patch to obtain the image patch feature vector. Multiple open object category feature vectors are obtained. Based on the image patch feature vector and the object category feature vector, the category of the image patch is determined, and the category of the image patch is used as the category of the object corresponding to the image patch. In this invention, image patches of objects are obtained through image segmentation and clustering, thus enabling the localization of open objects in the image. Finally, the category of the image patch is used as the category of the object corresponding to the image patch. Therefore, the category of open objects can be perceived. Furthermore, since the obtained open object categories are not specific, unspecified open object categories can be detected, improving the perception ability of open object categories.

[0058] The innovative aspects of this invention include:

[0059] 1. Image patches of objects are obtained through image segmentation and clustering. Therefore, it has the ability to locate the position of open objects in the image. Finally, the category of the image patch is used as the category of the object corresponding to the image patch. Therefore, the category of open object can be perceived. Furthermore, since the obtained open object category is not specific, it can detect unspecified open object categories, thus improving the perception ability of open object categories.

[0060] 2. For each point cloud in the point cloud sequence, based on the pose information of the LiDAR when the point cloud was acquired, as well as the intrinsic and extrinsic parameters of the camera, each point cloud point in the point cloud is projected onto the corresponding image frame to obtain the point cloud projection point, thereby realizing the correspondence between each point cloud point in the point cloud and each pixel point in the image frame.

[0061] 3. Determine the semantic edges contained in the graph based on the image segmentation results, and determine the geometric edges contained in the graph based on the Euclidean distance.

[0062] 4. By projecting the object into the image sequence, determine the minimum bounding rectangle region corresponding to the object in each image frame, and determine the image block corresponding to the object based on the minimum bounding rectangle region.

[0063] 5. By using a preset image segmentation method, all objects in the image frame can be distinguished. Then, by using graph optimization, point cloud points that are close to each other and belong to the same object can be found through geometric edges, and point cloud points that are far apart and belong to the same object can be found through semantic edges. Therefore, all objects can be found through geometric edges and semantic edges, which improves the recall rate of open objects.

[0064] 6. Since V is more related to the feature values ​​of pixels than Q and K, after changing the calculation method of the Transformer of the text encoder in the contrastive language-image pre-trained CLIP network from the QK attention weight matrix to the VV attention weight matrix, the feature values ​​of each pixel position can better integrate with their own related features. Therefore, it can improve the consistency between pixels and their semantics, thereby enhancing the reliability of category recognition.

[0065] 7. By using only the feature vectors projected from the point cloud of the object onto the image patch, interference from non-object pixel locations is reduced, thus improving the accuracy of category recognition.

[0066] 8. This invention utilizes the open object category recognition capability inherent in the contrastive language-image pre-trained CLIP network, which can identify open object categories that do not appear in the training data of the contrastive language-image pre-trained CLIP network.

[0067] 9. The calculation method of the Transformer of the text encoder in the contrastive language-image pre-trained CLIP network is modified from the QK attention weight matrix to the VV attention weight matrix. This allows the feature values ​​of each pixel position to better integrate with their own related features, thereby improving the consistency between pixels and their semantics. Furthermore, the feature vector of the image patch is obtained by projecting the point cloud points of the object corresponding to the image patch onto the image patch, rather than the feature vector of the entire image frame. Therefore, features that can better describe the object are obtained, thereby improving the accuracy of open object category recognition.

[0068] Of course, implementing any product or method of the present invention does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0069] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0070] Figure 1 A flowchart illustrating an open object category perception method for robot scenarios provided in an embodiment of the present invention;

[0071] Figure 2 This is a schematic diagram of a graph in graph optimization.

[0072] Figure 3 This is a schematic diagram of the structure of the contrastive language-image pre-trained CLIP network;

[0073] Figure 4 This is a schematic diagram of an open object category perception device for robot scenarios provided in an embodiment of the present invention. Detailed Implementation

[0074] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0075] It should be noted that the terms "comprising" and "having," and any variations thereof, in the embodiments and drawings of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0076] This invention discloses an open object category perception method and apparatus for robotic scenarios, which can improve the ability to perceive open object categories. The embodiments of this invention are described in detail below.

[0077] Figure 1 This is a flowchart illustrating an open object category perception method for robotic scenarios provided by an embodiment of the present invention. The method is applied to electronic devices. Specifically, the method includes the following steps.

[0078] S110: Obtain the point cloud sequence and the corresponding image sequence.

[0079] In this embodiment of the invention, the perception of open object categories consists of two steps:

[0080] first step:

[0081] Object recognition, that is, identifying objects in an image, but at this time the category of the object is unknown. This includes steps S120-S160.

[0082] Step Two:

[0083] Category identification, that is, obtaining the category of the open object from the identified objects, includes step S170.

[0084] To perceive the categories of open objects in an image, it is necessary to acquire point clouds using a LiDAR scanner on the vehicle and images using a camera, then obtain the point cloud sequence and the corresponding image sequence. Specifically, this can be done in real-time or offline. In real-time acquisition, the current image frame and a preset number of consecutive image frames preceding it, along with their corresponding point cloud sequences, can be acquired. In offline acquisition, any time period can be used to acquire the image sequence and its corresponding point cloud sequence.

[0085] S120: For each point cloud in the point cloud sequence, project each point cloud point in the point cloud onto the corresponding image frame according to the preset data preprocessing method to obtain the point cloud projection point.

[0086] In order to identify objects in an image frame, it is necessary to preprocess the point cloud sequence and the corresponding image sequence. Specifically, for each point cloud in the point cloud sequence, each point cloud point is projected onto the corresponding image frame according to a preset data preprocessing method to obtain the point cloud projection point.

[0087] Step S120 may include:

[0088] Acquire the intrinsic and extrinsic parameters of the camera that captured the image sequence, and the pose information of the LiDAR that acquired the point cloud sequence;

[0089] For each point cloud in the point cloud sequence, based on the pose information of the LiDAR when the point cloud was acquired, as well as the intrinsic and extrinsic parameters of the camera, each point cloud point in the point cloud is projected onto the corresponding image frame to obtain the point cloud projection point.

[0090] When the LiDAR on the vehicle acquires point clouds and the camera acquires images, the corresponding intrinsic and extrinsic parameters of the camera, as well as the pose information of the LiDAR, are also recorded, for example, in the vehicle's processor. Therefore, it is possible to obtain the intrinsic and extrinsic parameters of the camera that captured the image sequence and the pose information of the LiDAR that acquired the point cloud sequence.

[0091] Then, based on the pose information of the LiDAR and the intrinsic and extrinsic parameters of the camera, the point cloud sequence is projected into the image sequence. This yields the correspondence between each point in each point cloud within the point cloud sequence and each pixel in each image frame within the image sequence. The pixels in this correspondence are then used as point cloud projection points. In other words, for each point cloud in the point cloud sequence, based on the pose information of the LiDAR at the time of acquisition and the intrinsic and extrinsic parameters of the camera, each point in that point cloud is projected into the corresponding image frame to obtain the point cloud projection point. The method for projecting the point cloud sequence into the image sequence based on the LiDAR pose information and the camera's intrinsic and extrinsic parameters can be any existing projection method for projecting a point cloud sequence into an image sequence.

[0092] Therefore, for each point cloud in the point cloud sequence, based on the pose information of the LiDAR when the point cloud was acquired, as well as the intrinsic and extrinsic parameters of the camera, each point cloud point in the point cloud is projected onto the corresponding image frame to obtain the point cloud projection point, thus realizing the correspondence between each point cloud point in the point cloud and each pixel point in the image frame.

[0093] S130: For each image frame in the image sequence, the image frame is segmented according to the preset image segmentation method and all point cloud projection points in the image frame to obtain the image segmentation result of the image frame.

[0094] Since an image frame may contain a lot of content, such as trees, houses, and roads, it is necessary to perform image segmentation on each image frame to separate all the content in the image, so as to facilitate subsequent object recognition. Specifically, for each image frame in the image sequence, the image frame is segmented according to the preset image segmentation method and all point cloud projection points in the image frame to obtain the image segmentation result of the image frame.

[0095] The preset image segmentation method can be SAM (Segment Anything Model) or Mask2Former image segmentation model.

[0096] It should be noted that the image segmentation result obtained at this time only segments all the content in the image frame, which is unrelated to the category and does not contain semantic information.

[0097] S140: For each point cloud in the point cloud sequence, each point cloud point in the point cloud is used as a node of the graph, and the geometric edges and semantic edges contained in the graph are determined based on the image segmentation results of the image frame corresponding to the point cloud.

[0098] Since the image segmentation result only segments all the content in the image frame, some content belonging to the same object may be segmented into different image segmentation results. Therefore, it is not possible to locate the specific object. Therefore, in order to recognize the object, it is necessary to identify the specific object based on the image segmentation result. In this embodiment of the invention, the specific object is identified by graph optimization and clustering.

[0099] Specifically, the graph optimization method involves taking each point in the point cloud sequence as a node of the graph and determining the geometric and semantic edges contained in the graph based on the image segmentation results of the image frame corresponding to that point cloud.

[0100] Figure 2 This is a schematic diagram of a graph in graph optimization. See [link / reference]. Figure 2 In a graph optimization problem, there are nodes and edges. In this embodiment of the invention, for each point cloud in the point cloud sequence, each point cloud point is treated as a node of the graph, and two types of edges are defined: geometric edges and semantic edges. Figure 2 In the diagram, the dashed lines represent semantic edges, the dotted lines represent geometric edges, the circle with a value of 1 represents the node at time t-1, the circle with a value of 2 represents the node at time t, and the circle with a value of 3 represents the node at time t+1.

[0101] The determination of geometric and semantic edges in the graph based on the image segmentation results of the image frame corresponding to the point cloud may include:

[0102] Point cloud projection point pairs belonging to the same time step and the same image segmentation result are treated as semantic edges. The Euclidean distance between every two point cloud points in the graph is calculated. Point cloud point pairs belonging to different time steps and whose Euclidean distance is less than a preset distance threshold are treated as geometric edges. The preset distance threshold can be determined empirically.

[0103] In this embodiment of the invention, two point cloud points form an edge, that is, a pair of point cloud points forms an edge. A geometric edge means that the two point cloud points that make up the edge are close to each other, and a semantic edge means that the two point cloud points that make up the edge belong to the same image segmentation result at the same time.

[0104] See also Figure 2 There are a total of 6 semantic edges: A, B, C, D, E and F, and 3 geometric edges: G, H and J.

[0105] Therefore, semantic edges in the graph are determined based on image segmentation results, and geometric edges in the graph are determined based on Euclidean distance.

[0106] S150: Clustering is performed based on the preset clustering method, geometric edges, and semantic edges to obtain the clustering results of the objects in the graph.

[0107] After determining the geometric and semantic edges in the graph through graph optimization, clustering can be performed based on the preset clustering method, geometric edges, and semantic edges to obtain the clustering results of the objects in the graph.

[0108] The preset clustering method can be spectral clustering. Specifically, the Laplacian matrix of the graph is constructed based on geometric edges and semantic edges. The eigenvectors of the graph are obtained based on the Laplacian matrix, and then clustering is performed in the eigenvector space to obtain the clustering results of objects in the graph, that is, the specific objects in the graph are identified.

[0109] S160: For each object in the clustering results, project the object into the image sequence to obtain the corresponding image patch.

[0110] Step S160 identifies objects in the graph of the graph optimization. It is also necessary to locate the objects in the image frame based on this. Therefore, for each object in the clustering result, the object is projected onto the image sequence to obtain the image patch corresponding to the object.

[0111] The process of projecting the object onto the image sequence to obtain the corresponding image patch can include:

[0112] Based on the pose information of the lidar when collecting the point cloud corresponding to the object, as well as the intrinsic and extrinsic parameters of the camera, the object is projected into the image sequence, the minimum bounding rectangle region corresponding to the object in each image frame is determined, and the image block corresponding to the object is determined based on the minimum bounding rectangle region.

[0113] The method of projecting the object into the image sequence based on the pose information of the lidar and the intrinsic and extrinsic parameters of the camera when collecting the point cloud corresponding to the object is the same as the method of projecting the point cloud sequence into the image sequence based on the pose information of the lidar and the intrinsic and extrinsic parameters of the camera in step S120, and will not be repeated here.

[0114] After projecting the object into the image sequence, the minimum bounding rectangle region corresponding to the object in each image frame is taken, and the image block corresponding to the object is determined based on the minimum bounding rectangle region.

[0115] Since some objects may appear in multiple image frames, while others may appear in only one image frame, determining the image patch corresponding to the object based on the minimum bounding rectangle region can be done as follows:

[0116] Determine whether the object exists in an image frame of the image sequence. If so, take the smallest bounding rectangle region of the object in the existing image frame as the image block corresponding to the object.

[0117] If not, determine the smallest bounding rectangle region of the object in the image frame closest to the camera from the existing image frames as the image block corresponding to the object.

[0118] Since point clouds contain positional information, the distance between the image frame containing the object and the camera can be determined based on the point cloud points of the object.

[0119] At this point, we have only obtained the image patch corresponding to the object, but we still do not know what the category of the object is.

[0120] Therefore, by projecting the object into the image sequence, the minimum bounding rectangle region corresponding to the object in each image frame is determined, and the image block corresponding to the object is determined based on the minimum bounding rectangle region.

[0121] In this embodiment of the invention, all objects in an image frame can be distinguished by a preset image segmentation method. Then, a graph optimization method is used to find point cloud points that are close to each other and belong to the same object by using geometric edges and to find point cloud points that are far apart and belong to the same object by using semantic edges. Therefore, all objects can be found by using geometric edges and semantic edges, which improves the recall rate of open objects.

[0122] S170: For each image patch, input the image patch into a preset image classification network to obtain the feature map corresponding to the image patch, project the point cloud of the object corresponding to the image patch onto the image patch to obtain the image patch feature vector, obtain the object category feature vectors of multiple open object categories, determine the category of the image patch based on the image patch feature vector and the object category feature vector, and use the category of the image patch as the category of the object corresponding to the image patch.

[0123] To determine the category of an object, for each image patch, the image patch is input into a preset image classification network to obtain the feature map corresponding to that image patch.

[0124] The preset image classification network can be of various types, including but not limited to the following two: CLIP (Contrastive Language–Image Pre-training) network or OpenCLIP (Open Contrastive Language–Image Pre-training) network.

[0125] When the preset image classification network is a contrastive language-image pre-trained CLIP network, in one implementation, this embodiment of the invention improves the structure of the contrastive language-image pre-trained CLIP network to enhance the consistency between pixels and their semantics. The calculation method of the Transformer of the text encoder in the contrastive language-image pre-trained CLIP network is modified to the following formula:

[0126] Attn VV =softmax(V*V) T )*V

[0127] Among them, Attn VV Let V be the attention weight matrix, where V is the value, T is the transpose, and softmax is the normalization function.

[0128] The Transformer calculation method for the text encoder in the unmodified contrastive language-image pre-trained CLIP network is as follows:

[0129] Attn QK =softmax(Q*K) T )*V

[0130] Among them, Attn QK Let QK be the attention weight matrix, where V is the value, T is the transpose, Q is the query, K is the key, and softmax is the normalization function.

[0131] As can be seen from the two formulas above, since V is more related to the feature values ​​of pixels than Q and K, after changing the calculation method of the Transformer of the text encoder in the contrastive language-image pre-trained CLIP network from the QK attention weight matrix to the VV attention weight matrix, the feature values ​​of each pixel position can better integrate with their own related features. Therefore, it can improve the consistency between pixels and their semantics, thereby enhancing the reliability of category recognition.

[0132] Figure 3 For a schematic diagram of the contrast language-image pre-trained CLIP network, see [link / reference]. Figure 3 When the preset image classification network is the contrastive language-image pre-trained CLIP network, the image block is input into the image encoder of the contrastive language-image pre-trained CLIP network to obtain the feature map M corresponding to the image block: (1+h*w)*C, where C is the feature dimension, h is the width of the image block, and w is the height of the image block. For example, C is 512, h is 224, and w is 224.

[0133] After obtaining the feature map corresponding to the image patch, the point cloud points of the object corresponding to the image patch are projected onto the image patch to obtain the image patch feature vector.

[0134] The process of projecting the point cloud points of the object corresponding to the image patch onto the image patch to obtain the image patch feature vector can include:

[0135] The point cloud points of the object corresponding to the image patch are projected onto the image patch to obtain multiple feature vectors. The average value of the multiple feature vectors is calculated as the image patch feature vector of the image patch.

[0136] Specifically, the projection method of projecting the point cloud of the object onto the image block is the same as the method of projecting the point cloud sequence onto the image sequence based on the pose information of the lidar and the intrinsic and extrinsic parameters of the camera in step S120, and will not be repeated here.

[0137] Therefore, by using only the feature vectors projected from the point cloud of the object onto the image patch, interference from non-object pixel locations is reduced, thus improving the accuracy of category recognition.

[0138] After obtaining the image patch feature vector, object category feature vectors for multiple open object categories are obtained. The category of the image patch is determined based on the image patch feature vector and the object category feature vector, and the category of the image patch is used as the category of the object corresponding to the image patch.

[0139] This process, which involves obtaining object category feature vectors for multiple open object categories and determining the category of an image patch based on the image patch feature vector and the object category feature vector, may include:

[0140] Based on the category names of multiple open object categories and a preset feature vector extraction algorithm, the object category feature vector for each open object category is obtained;

[0141] Calculate the similarity between each object category feature vector and the image patch feature vector, and take the open object category corresponding to the object category feature vector with the highest similarity as the category of the image patch.

[0142] When the preset feature vector extraction algorithm is a contrastive language-image pre-trained CLIP network, the object category feature vector for each open object category obtained based on the category names of multiple open object categories and the preset feature vector extraction algorithm can be as follows:

[0143] The category names of multiple open object categories are input into the text encoder in the contrastive language-image pre-trained CLIP network to obtain the object category feature vector for each open object category.

[0144] See also Figure 3The category names of multiple open object categories include at least: vehicle, garbage, fire hydrant, and stone. After being processed by an image template, the category names of multiple open object categories are input into the text encoder in the contrastive language-image pre-trained CLIP network to obtain the object category feature vectors of N open object categories, where R1 represents the object category feature vector of the first open object category, R2 represents the object category feature vector of the second open object category, and RN represents the object category feature vector of the Nth open object category.

[0145] After obtaining the object category feature vector for each open object category, the similarity between each object category feature vector and the image patch feature vector can be calculated. The open object category corresponding to the object category feature vector with the highest similarity is taken as the category of that image patch. The similarity can be calculated using the cosine similarity method.

[0146] See also Figure 3 MR1 calculates the similarity between the image patch feature vector and the object category feature vector of the first open object category; MR2 calculates the similarity between the image patch feature vector and the object category feature vector of the second open object category; MR3 calculates the similarity between the image patch feature vector and the object category feature vector of the third open object category; and MRN calculates the similarity between the image patch feature vector and the object category feature vector of the Nth open object category. Finally, the open object category corresponding to the object category feature vector with the highest similarity is fire hydrant, so fire hydrant is taken as the category of the image patch.

[0147] Therefore, this invention utilizes the open object category recognition capability inherent in the contrastive language-image pre-trained CLIP network to identify open object categories that do not appear in the training data of the contrastive language-image pre-trained CLIP network.

[0148] In this embodiment of the invention, the calculation method of the Transformer of the text encoder in the contrastive language-image pre-trained CLIP network is modified from the QK attention weight matrix to the VV attention weight matrix, so that the feature value of each pixel position can better integrate with its own related features. Therefore, the consistency between the pixel and its semantics can be improved. Furthermore, the point cloud points of the object corresponding to the image patch are projected onto the image patch to obtain the image patch feature vector, which is not the feature vector of the entire image frame. Therefore, features that can better describe the object are obtained, thereby improving the accuracy of open object category recognition.

[0149] As can be seen from the above, this embodiment can obtain a point cloud sequence and a corresponding image sequence. For each point cloud in the point cloud sequence, each point cloud point is projected onto the corresponding image frame according to a preset data preprocessing method to obtain point cloud projection points. For each image frame in the image sequence, the image frame is segmented according to a preset image segmentation method and all point cloud projection points in the image frame to obtain the image segmentation result of the image frame. For each point cloud in the point cloud sequence, each point cloud point is used as a node of the graph. Based on the image segmentation result of the image frame corresponding to the point cloud, the geometric edges and semantic edges contained in the graph are determined. Clustering is performed using a pre-defined clustering method, geometric edges, and semantic edges to obtain clustering results for objects in the image. For each object in the clustering results, the object is projected onto the image sequence to obtain the corresponding image patch. For each image patch, the image patch is input into a pre-defined image classification network to obtain the corresponding feature map. The point cloud points of the object corresponding to the image patch are projected onto the image patch to obtain the image patch feature vector. Multiple open object category feature vectors are obtained. Based on the image patch feature vector and the object category feature vector, the category of the image patch is determined, and the category of the image patch is used as the category of the object corresponding to the image patch. In this invention, image patches of objects are obtained through image segmentation and clustering, thus enabling the localization of open objects in the image. Finally, the category of the image patch is used as the category of the object corresponding to the image patch. Therefore, the category of open objects can be perceived. Furthermore, since the obtained open object categories are not specific, unspecified open object categories can be detected, improving the perception capability of open object categories.

[0150] Figure 4 This is a schematic diagram of an open object category perception device for robotic scenarios provided in an embodiment of the present invention. See also... Figure 4 The present invention provides an open object category perception device for robot scenarios, which may include:

[0151] The acquisition module 410 is used to acquire point cloud sequences and corresponding image sequences;

[0152] The preprocessing module 420 is used to project each point cloud in the point cloud sequence onto the corresponding image frame to obtain the point cloud projection point according to a preset data preprocessing method.

[0153] The image segmentation module 430 is used to segment each image frame in the image sequence according to a preset image segmentation method and all point cloud projection points in the image frame, and obtain the image segmentation result of the image frame.

[0154] The graph optimization module 440 is used to, for each point cloud in the point cloud sequence, take each point cloud point in the point cloud as a node of the graph, and determine the geometric edges and semantic edges contained in the graph based on the image segmentation result of the image frame corresponding to the point cloud.

[0155] Clustering module 450 is used to perform clustering based on a preset clustering method, the geometric edges, and the semantic edges to obtain the clustering results of the objects in the graph;

[0156] The projection module 460 is used to project each object in the clustering result onto the image sequence to obtain the image patch corresponding to the object;

[0157] The category perception module 470 is used to input the image block into a preset image classification network for each image block to obtain the feature map corresponding to the image block, project the point cloud points of the object corresponding to the image block onto the image block to obtain the image block feature vector, obtain the object category feature vectors of multiple open object categories, determine the category of the image block based on the image block feature vector and the object category feature vector, and use the category of the image block as the category of the object corresponding to the image block.

[0158] Therefore, the device provided in this embodiment can acquire point cloud sequences and corresponding image sequences. For each point cloud in the point cloud sequence, each point cloud point is projected onto the corresponding image frame according to a preset data preprocessing method to obtain point cloud projection points. For each image frame in the image sequence, the image frame is segmented according to a preset image segmentation method and all point cloud projection points in the image frame to obtain the image segmentation result of the image frame. For each point cloud in the point cloud sequence, each point cloud point is used as a node of the graph. Based on the image segmentation result of the image frame corresponding to the point cloud, the geometric edges and semantic edges contained in the graph are determined. Clustering is performed using a pre-defined clustering method, geometric edges, and semantic edges to obtain clustering results for objects in the image. For each object in the clustering results, the object is projected onto the image sequence to obtain the corresponding image patch. For each image patch, the image patch is input into a pre-defined image classification network to obtain the corresponding feature map. The point cloud points of the object corresponding to the image patch are projected onto the image patch to obtain the image patch feature vector. Multiple open object category feature vectors are obtained. Based on the image patch feature vector and the object category feature vector, the category of the image patch is determined, and the category of the image patch is used as the category of the object corresponding to the image patch. In this invention, image patches of objects are obtained through image segmentation and clustering, thus enabling the localization of open objects in the image. Finally, the category of the image patch is used as the category of the object corresponding to the image patch. Therefore, the category of open objects can be perceived. Furthermore, since the obtained open object categories are not specific, unspecified open object categories can be detected, improving the perception capability of open object categories.

[0159] In one implementation, the preprocessing module 420 may include:

[0160] The acquisition submodule is used to acquire the intrinsic and extrinsic parameters of the camera that captures the image sequence and the pose information of the lidar that collects the point cloud sequence.

[0161] The projection submodule is used to project each point cloud in the point cloud sequence onto the corresponding image frame to obtain the point cloud projection point, based on the pose information of the lidar when the point cloud was acquired and the intrinsic and extrinsic parameters of the camera.

[0162] In one implementation, the preset image segmentation method can be either the arbitrary segmentation model SAM or the Mask2Former image segmentation model.

[0163] In one implementation, the graph optimization module 440 can be specifically used for:

[0164] Point cloud point pairs corresponding to point cloud projection point pairs belonging to the same time and the same image segmentation result are taken as a semantic edge. The Euclidean distance between every two point cloud points in the graph is calculated. Point cloud point pairs belonging to different time and whose Euclidean distance is less than a preset distance threshold are taken as a geometric edge.

[0165] In one implementation, the preset clustering method can be a spectral clustering method.

[0166] In one implementation, the projection module 460 can be specifically used for:

[0167] Based on the pose information of the lidar when collecting the point cloud corresponding to the object, as well as the intrinsic and extrinsic parameters of the camera, the object is projected into the image sequence, the minimum bounding rectangle region corresponding to the object in each image frame is determined, and the image block corresponding to the object is determined based on the minimum bounding rectangle region.

[0168] In one implementation, the preset image classification network can be a contrastive language-image pre-trained CLIP network or an open contrastive language-image pre-trained OpenCLIP network.

[0169] In one implementation, the Transformer of the text encoder in the contrastive language-image pre-trained CLIP network can be computed as follows:

[0170] Attn VV =softmax(V*V) T )*V

[0171] Among them, Attn VV Let V be the attention weight matrix, where V is the value, T is the transpose, and softmax is the normalization function.

[0172] In one implementation, the category-aware module 470 may include:

[0173] The feature extraction submodule is used to obtain the object category feature vector for each open object category based on the category names of multiple open object categories and a preset feature vector extraction algorithm.

[0174] The calculation submodule is used to calculate the similarity between each object category feature vector and the image patch feature vector, and to take the open object category corresponding to the object category feature vector with the highest similarity as the category of the image patch.

[0175] The above-described apparatus embodiments correspond to the method embodiments and have the same technical effects. For detailed explanations, please refer to the method embodiments. The apparatus embodiments are derived from the method embodiments; detailed explanations can be found in the method embodiments section, and will not be repeated here.

[0176] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing the present invention.

[0177] Those skilled in the art will understand that the modules in the apparatus of the embodiments can be distributed in the apparatus of the embodiments as described in the embodiments, or they can be located in one or more devices different from this embodiment with corresponding changes. The modules of the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0178] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An open object category perception method for robot scenarios, characterized in that, include: Obtain the point cloud sequence and the corresponding image sequence; For each point cloud in the point cloud sequence, each point cloud point is projected onto the corresponding image frame according to a preset data preprocessing method to obtain the point cloud projection point; For each image frame in the image sequence, the image frame is segmented according to a preset image segmentation method and all point cloud projection points in the image frame to obtain the image segmentation result of the image frame; For each point cloud in the point cloud sequence, each point in the point cloud is taken as a node of the graph. Based on the image segmentation result of the image frame corresponding to the point cloud, the geometric edges and semantic edges contained in the graph are determined. The step of determining the geometric edges and semantic edges contained in the graph based on the image segmentation result of the image frame corresponding to the point cloud includes: Point cloud point pairs corresponding to point cloud projection point pairs belonging to the same time and the same image segmentation result are taken as a semantic edge. The Euclidean distance between every two point cloud points in the graph is calculated. Point cloud point pairs belonging to different times and whose Euclidean distance is less than a preset distance threshold are taken as a geometric edge. Clustering is performed based on a preset clustering method, the geometric edges, and the semantic edges to obtain the clustering results of the objects in the graph; For each object in the clustering result, the object is projected onto the image sequence to obtain the image patch corresponding to the object; For each image patch, the image patch is input into a preset image classification network to obtain the feature map corresponding to the image patch. The point cloud points of the object corresponding to the image patch are projected onto the image patch to obtain the image patch feature vector. Multiple open object category feature vectors are obtained. The category of the image patch is determined based on the image patch feature vector and the object category feature vector, and the category of the image patch is used as the category of the object corresponding to the image patch.

2. The method as described in claim 1, characterized in that, The step of projecting each point cloud in the point cloud sequence onto a corresponding image frame to obtain the projected point cloud points according to a preset data preprocessing method includes: Acquire the intrinsic and extrinsic parameters of the camera that captured the image sequence and the pose information of the lidar that acquired the point cloud sequence; For each point cloud in the point cloud sequence, based on the pose information of the lidar when the point cloud was acquired and the intrinsic and extrinsic parameters of the camera, each point cloud point in the point cloud is projected into the corresponding image frame to obtain the point cloud projection point.

3. The method as described in claim 1, characterized in that, The preset image segmentation method is either the SAM or Mask2Former image segmentation model.

4. The method as described in claim 1, characterized in that, The preset clustering method is the spectral clustering method.

5. The method as described in claim 2, characterized in that, The step of projecting the object onto the image sequence to obtain the image patch corresponding to the object includes: Based on the pose information of the lidar when collecting the point cloud corresponding to the object, as well as the intrinsic and extrinsic parameters of the camera, the object is projected into the image sequence, the minimum bounding rectangle region corresponding to the object in each image frame is determined, and the image block corresponding to the object is determined based on the minimum bounding rectangle region.

6. The method as described in claim 1, characterized in that, The preset image classification network is either a contrastive language-image pre-trained CLIP network or an open contrastive language-image pre-trained OpenCLIP network.

7. The method as described in claim 6, characterized in that, The Transformer of the text encoder in the contrastive language-image pre-trained CLIP network is calculated as follows: ； in, Let V be the attention weight matrix, where V is the value and T is the transpose. This is the normalization function.

8. The method as described in claim 1, characterized in that, The step of obtaining object category feature vectors for multiple open object categories, and determining the category of an image patch based on the image patch feature vector and the object category feature vector, includes: Based on the category names of multiple open object categories and a preset feature vector extraction algorithm, the object category feature vector for each open object category is obtained; Calculate the similarity between each object category feature vector and the image patch feature vector, and take the open object category corresponding to the object category feature vector with the highest similarity as the category of the image patch.

9. An open object category perception device for robot scenarios, characterized in that, include: The acquisition module is used to acquire point cloud sequences and corresponding image sequences; The preprocessing module is used to project each point cloud in the point cloud sequence onto the corresponding image frame to obtain the point cloud projection point according to a preset data preprocessing method. The image segmentation module is used to segment each image frame in the image sequence according to a preset image segmentation method and all point cloud projection points in the image frame, so as to obtain the image segmentation result of the image frame. The graph optimization module is used to, for each point cloud in the point cloud sequence, treat each point cloud point as a node of the graph, and determine the geometric edges and semantic edges contained in the graph based on the image segmentation results of the image frame corresponding to the point cloud. Specifically, the graph optimization module is used to: Point cloud point pairs corresponding to point cloud projection point pairs belonging to the same time and the same image segmentation result are taken as a semantic edge. The Euclidean distance between every two point cloud points in the graph is calculated. Point cloud point pairs belonging to different times and whose Euclidean distance is less than a preset distance threshold are taken as a geometric edge. The clustering module is used to perform clustering based on a preset clustering method, the geometric edges, and the semantic edges to obtain the clustering results of the objects in the graph; The projection module is used to project each object in the clustering result onto the image sequence to obtain the image patch corresponding to the object; The category-aware module is used to input each image patch into a preset image classification network to obtain the feature map corresponding to the image patch, project the point cloud points of the object corresponding to the image patch onto the image patch to obtain the image patch feature vector, obtain the object category feature vectors of multiple open object categories, determine the category of the image patch based on the image patch feature vector and the object category feature vector, and use the category of the image patch as the category of the object corresponding to the image patch.

10. The apparatus as claimed in claim 9, characterized in that, The preprocessing module includes: The acquisition submodule is used to acquire the intrinsic and extrinsic parameters of the camera that captures the image sequence and the pose information of the lidar that collects the point cloud sequence. The projection submodule is used to project each point cloud in the point cloud sequence onto the corresponding image frame to obtain the point cloud projection point, based on the pose information of the lidar when the point cloud was acquired and the intrinsic and extrinsic parameters of the camera.

11. The apparatus as claimed in claim 9, characterized in that, The preset image segmentation method is either the SAM or Mask2Former image segmentation model.

12. The apparatus as claimed in claim 9, characterized in that, The preset clustering method is the spectral clustering method.

13. The apparatus as claimed in claim 10, characterized in that, The projection module is specifically used for: Based on the pose information of the lidar when collecting the point cloud corresponding to the object, as well as the intrinsic and extrinsic parameters of the camera, the object is projected into the image sequence, the minimum bounding rectangle region corresponding to the object in each image frame is determined, and the image block corresponding to the object is determined based on the minimum bounding rectangle region.

14. The apparatus as claimed in claim 9, characterized in that, The preset image classification network is either a contrastive language-image pre-trained CLIP network or an open contrastive language-image pre-trained OpenCLIP network.

15. The apparatus as claimed in claim 14, characterized in that, The Transformer of the text encoder in the contrastive language-image pre-trained CLIP network is calculated as follows: ； in, Let V be the attention weight matrix, where V is the value and T is the transpose. This is the normalization function.

16. The apparatus as claimed in claim 9, characterized in that, The category-aware module includes: The feature extraction submodule is used to obtain the object category feature vector for each open object category based on the category names of multiple open object categories and a preset feature vector extraction algorithm. The calculation submodule is used to calculate the similarity between each object category feature vector and the image patch feature vector, and to take the open object category corresponding to the object category feature vector with the highest similarity as the category of the image patch.

Images