Cross-modal scene matching method, device and equipment for laser point cloud and visual image

Abstract

The embodiments of the present application provide a method, device and equipment for cross-modal scene matching of laser point clouds and visual images. The method includes: semantic segmentation of the image to be processed obtained by shooting the target vehicle during driving to obtain a semantic image; clustering the semantic image to obtain the corresponding visual graph node; obtaining a pre-built offline point cloud graph node map library; based on a graph convolutional neural network, respectively extracting the visual graph node aggregation features corresponding to the visual graph node and the point cloud graph node aggregation features corresponding to each frame of point cloud graph node data; splicing the visual graph node aggregation features and the point cloud graph node aggregation features, and inputting the spliced ​​features into a fully connected network, so that the fully connected network outputs the matching degree corresponding to the two modalities. The technical solution of the embodiment of the present application reduces the complexity of the cross-modal matching calculation and ensures the accuracy of the calculation results of the cross-modal matching.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, apparatus and device for cross-modal scene matching of laser point clouds and visual images. Background Art

[0002] The advancement of autonomous driving technology is placing higher demands on the stability and reliability of autonomous vehicle positioning technology. Traditional autonomous vehicle positioning often relies solely on lidar or vision technology. However, a single sensor alone struggles to achieve accurate positioning in complex urban environments. Cross-modal technology leverages the complementary strengths of multiple sensors to achieve high-precision vehicle positioning.

[0003] To achieve cross-modal matching between two sensors, it is necessary to resolve the differences between the laser point cloud and the visual image modality. Current technical solutions often use raw laser point cloud and visual image point cloud information for cross-modal feature fusion. However, using only raw data information can lead to the loss of important information and increase computational complexity. Therefore, how to reduce the complexity of cross-modal matching calculations and ensure the accuracy of cross-modal matching calculation results has become a technical problem that needs to be solved urgently. Summary of the Invention

[0004] The embodiments of the present application provide a method, apparatus, and device for cross-modal scene matching of laser point clouds and visual images, which can reduce the complexity of cross-modal matching calculations to at least a certain extent and ensure the accuracy of the calculation results of cross-modal matching.

[0005] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by practice of the present application.

[0006] According to one aspect of an embodiment of the present application, a cross-modal scene matching method for laser point clouds and visual images is provided, the method comprising:

[0007] Acquire the image to be processed that is taken while the target vehicle is driving;

[0008] Performing semantic segmentation on the image to be processed to determine the semantic category corresponding to each pixel in the image to be processed to obtain a semantic image;

[0009] Clustering the semantic image to obtain corresponding visual graph nodes;

[0010] Obtain a pre-built offline point cloud map node library, wherein the offline point cloud map node library includes a plurality of frames of point cloud map node data;

[0011] Based on the graph convolutional neural network, the visual graph node aggregation features corresponding to the visual graph nodes and the point cloud graph node aggregation features corresponding to the point cloud graph node data of each frame are respectively extracted;

[0012] The visual graph node aggregation features and the point cloud graph node aggregation features are spliced, and the spliced ​​features are input into a fully connected network so that the fully connected network outputs the matching degree corresponding to the two modalities.

[0013] According to one aspect of an embodiment of the present application, a cross-modal scene matching device for laser point clouds and visual images is provided, comprising:

[0014] An image acquisition module is used to obtain images to be processed taken by the target vehicle during its driving process;

[0015] a first semantic segmentation module, configured to perform semantic segmentation on the image to be processed, and determine a semantic category corresponding to each pixel in the image to be processed, so as to obtain a semantic image;

[0016] An image clustering module, used to cluster the semantic images to obtain corresponding visual graph nodes;

[0017] A map acquisition module is used to acquire a pre-built offline point cloud map node map library, wherein the offline point cloud map node map library includes several frames of point cloud map node data;

[0018] A feature extraction module is used to extract the visual graph node aggregation features corresponding to the visual graph nodes and the point cloud graph node aggregation features corresponding to the point cloud graph node data of each frame based on a graph convolutional neural network;

[0019] The first processing module is used to splice the visual graph node aggregation features and the point cloud graph node aggregation features, and input the spliced ​​features into a fully connected network so that the fully connected network outputs the matching degree corresponding to the two modalities.

[0020] According to one aspect of an embodiment of the present application, a computer-readable medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the cross-modal scene matching method of laser point cloud and visual image as described in the above embodiment is implemented.

[0021] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the cross-modal scene matching method of laser point clouds and visual images as described in the above embodiments.

[0022] According to one aspect of an embodiment of the present application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the cross-modal scene matching method for laser point clouds and visual images provided in the above-described embodiments.

[0023] In the technical solutions provided in some embodiments of the present application, by obtaining an image to be processed taken while the target vehicle is driving, semantic segmentation is performed on the image to be processed, the semantic category corresponding to each pixel in the image to be processed is determined, a semantic image is obtained, the semantic image is clustered, the corresponding visual graph node is obtained, and then a pre-built offline point cloud graph node map library is obtained. The offline point cloud graph node map library includes several frames of point cloud graph node data. Based on the graph convolutional neural network, the visual graph node aggregation features corresponding to the visual graph node and the point cloud graph node aggregation features corresponding to each frame of point cloud graph node data are extracted respectively. The visual graph node aggregation features and the point cloud graph node aggregation features are spliced, and the spliced ​​features are input into a fully connected network to output the matching degree corresponding to the two modalities. Therefore, by pre-building an offline point cloud graph node map library, the processing process of the laser point cloud data is reduced in the actual matching process, the complexity of the cross-modal matching calculation is reduced, and the accuracy of the calculation results of the cross-modal matching is guaranteed.

[0024] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings are incorporated into and constitute a part of the specification, illustrating embodiments consistent with the present application and, together with the specification, explaining the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can derive other drawings based on these drawings without inventive effort. In the drawings:

[0026] Figure 1 A schematic diagram of a process for a cross-modal scene matching method of a laser point cloud and a visual image according to an embodiment of the present application is shown;

[0027] Figure 2 A schematic diagram of a semantic image according to an embodiment of the present application is shown;

[0028] Figure 3 A schematic diagram of the structure of a graph convolutional neural network according to an embodiment of the present application is shown;

[0029] Figure 4 A block diagram of a cross-modal scene matching device for laser point clouds and visual images according to an embodiment of the present application is shown;

[0030] Figure 5 A schematic diagram of the structure of a computer system suitable for implementing an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0031] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art.

[0032] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner.In the following description, many specific details are provided so as to provide a full understanding of the embodiments of the present application. However, it will be appreciated by those skilled in the art that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps etc. can be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0033] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0034] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0035] Figure 1 This figure shows a flow chart of a cross-modal scene matching method for laser point clouds and visual images according to one embodiment of the present application. This method can be applied to a terminal device or server, where the terminal device includes one or more of a smartphone, tablet computer, laptop computer, desktop computer, and vehicle-mounted terminal. The server can be a physical server or a cloud server.

[0036] The following uses this method applied to a vehicle terminal as an example for explanation. Figure 1 The method at least includes steps S110 to S160, which are described in detail as follows:

[0037] In step S110, an image to be processed is obtained which is captured while the target vehicle is traveling.

[0038] In this embodiment, the target vehicle may be equipped with an image acquisition device, such as a camera, which can capture images in front of the target vehicle in real time while the vehicle is in motion. The image acquisition device is communicatively connected to the vehicle terminal and can transmit the acquired images as images to be processed to the vehicle terminal in real time.

[0039] In step S120 , semantic segmentation is performed on the image to be processed to determine the semantic category corresponding to each pixel in the image to be processed, so as to obtain a semantic image.

[0040] In this embodiment, the vehicle terminal can perform semantic segmentation on the received image to be processed to extract the semantic features of each pixel in the image to be processed, and assign a corresponding semantic category to each pixel based on the semantic features corresponding to each pixel to obtain a semantic image.

[0041] In one embodiment, the vehicle terminal can call the DeepLab series of algorithms to perform semantic segmentation on the image to be processed, so as to assign a corresponding semantic category to each pixel point and obtain a semantic image. In one example, considering that the laser point cloud and the visual image have different modal characteristics, there will be differences in the classification of certain categories. To this end, the image to be processed can be integrated based on the laser point cloud and dynamic categories such as pedestrians can be filtered, so that a semantic image with the same classification as the semantic category of the laser point cloud can be output (such as Figure 2 shown).

[0042] In other words, because the semantic categories obtained from laser point clouds are relatively broad, such as "vehicle," "building," and "pedestrian," while the semantic categories obtained from visual images are relatively detailed, such as "vehicle" and "car," the "processed image can be based on the laser point cloud" mentioned in this application means unifying the semantic categories of the visual image (such as "truck" and "car") into the semantic category "vehicle" based on the laser point cloud; and directly filtering out dynamic categories such as "pedestrian" and not displaying them in the semantic image. (This summary is also acceptable.)

[0043] In step S130, the semantic image is clustered to obtain corresponding visual graph nodes.

[0044] In this embodiment, the vehicle-mounted terminal may cluster and divide each pixel in the semantic image according to the semantic category corresponding to the pixel, thereby obtaining the corresponding visual graph node.

[0045] In one embodiment of the present application, clustering the semantic image to obtain corresponding visual graph nodes includes:

[0046] Determining depth information of each pixel in the semantic image;

[0047] According to the depth information and the semantic category corresponding to each pixel, connected pixels with the same semantic category are divided into the same visual instance;

[0048] The mean value of the pixel coordinate values, the average depth value, and the semantic category in each of the visual instances are stored as the corresponding visual graph node.

[0049] In this embodiment, the vehicle terminal can call a pre-trained image clustering algorithm to cluster pixels with the same semantic category (for example, the same color) into visual graph nodes. Specifically, a binocular stereo matching algorithm can be used to calculate the depth information of each pixel in the semantic image. Based on the depth information of each pixel and the semantic category corresponding to each pixel, the vehicle terminal can divide the connected pixels with the same semantic category in the semantic image into the same visual instance to ensure the accuracy of the divided visual instances and obtain richer semantic image scene information.

[0050] Then, the vehicle-mounted terminal can determine the mean coordinate value, average depth value and semantic category of the pixel points in the same visual instance, and store them as the instantiation result of the visual graph node corresponding to the visual instance.

[0051] In one embodiment, during the clustering process, in order to avoid clustering semantic noise information into a visual graph node, a pixel number threshold can be determined according to actual implementation needs to filter noise data, thereby weakening the impact of noise on overall performance accuracy.

[0052] Please continue to refer to Figure 1 In step S140, a pre-built offline point cloud map node map library is obtained, and the offline point cloud map node map library includes several frames of point cloud map node data.

[0053] In this embodiment, those skilled in the art can pre-build and store an offline point cloud map node library for acquisition by the vehicle terminal. The offline point cloud map node library includes several frames of point cloud map node data. It should be understood that the offline point cloud map node library should correspond to the current driving section of the target vehicle or contain point cloud data corresponding to the current driving section of the target vehicle, that is, those skilled in the art can pre-collect the laser point cloud data of the current driving section and perform corresponding processing to obtain the corresponding offline point cloud map node library.

[0054] In one embodiment of the present application, building an offline point cloud node map library includes:

[0055] Obtaining original point cloud data for the target road section;

[0056] Performing semantic segmentation on the original point cloud data to determine the semantic category corresponding to each point in the original point cloud data to obtain a semantic point cloud;

[0057] Clustering is performed based on the semantic category corresponding to each point in the semantic point cloud and the distance between points, determining point cloud instances in the semantic point cloud, and determining the point cloud graph nodes corresponding to each point cloud instance;

[0058] Based on the point cloud node data of each frame, an offline point cloud node map library is constructed.

[0059] In this embodiment, a laser radar installed on a vehicle can be used to acquire raw point cloud data of the target road section, and semantic segmentation can be performed on the acquired raw point cloud data to determine the semantic category corresponding to each point in the raw point cloud, thereby obtaining a semantic point cloud. In one example, the RangeNet++ laser point cloud segmentation algorithm model can be used to extract semantic features from the raw point cloud data, thereby determining the corresponding semantic category for each point in the raw point cloud.

[0060] Next, clustering is performed based on the semantic category corresponding to each point in the semantic point cloud and the distance between points, and then the points in the semantic point cloud are divided into at least one point cloud instance to determine the corresponding point cloud graph node according to the point cloud instance.

[0061] In one embodiment of the present application, clustering is performed based on the semantic category corresponding to each point in the semantic point cloud and the distance between points, determining point cloud instances in the semantic point cloud, and determining the point cloud graph nodes corresponding to each point cloud instance, including:

[0062] According to the semantic category corresponding to each point in the semantic point cloud, the points in the semantic point cloud are divided into point sets corresponding to each semantic category;

[0063] For each point in the point set, search for points in the point set within its corresponding neighborhood. If the number of points found is greater than or equal to a predetermined threshold, add the points in the neighborhood to a search queue and continue searching for the points in the search queue.

[0064] Divide the searched points into the same point cloud instance, and determine the centroid information corresponding to the point cloud instance based on the position information of the points in the point cloud instance;

[0065] The semantic category and centroid information corresponding to the point cloud instance are stored as the point cloud graph node corresponding to the point cloud instance.

[0066] In this embodiment, the points in the semantic point cloud can be divided into point sets S={s1, s2, ..., s l}, where l is the number of the semantic category. A kd-tree (k-dimensional tree structure) can be established for each point set to store the laser point cloud data. For each point p in the point set i , use the kd-tree of its corresponding category to search for points in its neighborhood. If the number of points found is greater than or equal to the predetermined threshold Thread(p i ), then the points in the neighborhood are added to the search queue.

[0067] Next, each point in the search queue is searched. It should be understood that points that meet the above conditions obtained in subsequent searches are also added to the search queue for search until all points in the search queue are calculated. After all points are calculated, all traversed points are divided into the same point cloud instance I.

[0068] Repeat the above steps until all points in the point set are calculated to complete the instantiation process of the single-frame point cloud data.

[0069] For example, the number of points to be clustered is N. First, determine the farthest point in the point cloud and set it as p max , thus obtaining the maximum distance d in the point cloud data max For any point p in the point cloud i , calculate the Euclidean distance between the point and the origin of the coordinate system, set it to d, and finally get each point p i The corresponding clustering radius (ie neighborhood radius) when clustering is Where a and r are adjustment coefficients related to point cloud density and semantic category. We further set the Euclidean distance threshold, that is, two points p in the same category i and p j If there is a distance between dist(p i ,p j )≤R(p i ), then p i and p j Aggregate into the same cluster to form the same point cloud instance.

[0070] In one example, the predetermined threshold Thread(p i ) is calculated as follows:

[0071]

[0072] Where T is an adjustment coefficient. Thus, the predetermined threshold value can be adaptively changed based on the relevant features of the points to be searched, thereby ensuring the accuracy of the subsequent instance segmentation results. In other examples, the predetermined threshold value can also be set as a fixed value by those skilled in the art based on prior experience, and this is not particularly limited.

[0073] When the point cloud instance is determined, let SL = {L1, L2, ..., L} represent the instantiated point cloud set, where L = {I1, I2, ..., I N} means dividing the entire point cloud of this frame into different semantic categories and obtaining all instances of the set under each semantic category. Then, the center of mass information corresponding to the point cloud instance is determined by taking the average value of the coordinate information of the points in each point cloud instance. Then the semantic category l and centroid information corresponding to the point cloud instance Store it as the point cloud graph node corresponding to the point cloud instance.

[0074] Then, the point cloud node data of each frame can be integrated to build an offline point cloud node map library.

[0075] Therefore, the embodiment provided by the present application first uses laser point cloud data to construct an offline point cloud map node map library. In the actual positioning process, only visual images are used for matching, that is, visual images are used to search with the offline point cloud map node map library. Therefore, in the map library construction stage, the vehicle only needs to be equipped with a laser radar, and in the real-time matching stage, the vehicle only needs an image acquisition device to work, which not only reduces the complexity of the cross-modal matching calculation, but also ensures the accuracy of the cross-modal matching calculation results.

[0076] Please continue to refer to Figure 1 In step S150, based on the graph convolutional neural network, the visual graph node aggregation features corresponding to the visual graph nodes and the point cloud graph node aggregation features corresponding to the point cloud graph node data of each frame are extracted respectively.

[0077] In this embodiment, the vehicle terminal can be based on the EdgeConv structure of the graph convolutional neural network (such as Figure 3 As shown in Figure 2, it is used to perform feature aggregation and feature encoding on graph nodes to capture the feature information of graph nodes. Based on the graph convolutional neural network, each graph node can be aggregated, convolved, and pooled with its surrounding neighbors to obtain the aggregated features of each graph node. The aggregated features of the graph nodes are further input into the attention module to obtain the global aggregated features of the graph nodes.

[0078] Specifically, set X i is the semantic feature information of the graph node, where i represents any node. Further, the adjacent k graph nodes are defined as Xij , where j∈{1,2,…,k}, the aggregated features at this point are expressed as follows:

[0079]

[0080] Among them, hΘ(·) represents the aggregation function, which is used to aggregate X i Global information and X i With X ij The local information between them is used to extract the features of the graph nodes, and the extracted node aggregation features are defined as n i ∈R D . In calculating the global average feature f avg When additional learnable matrix M∈R D×D , to dynamically adjust the impact of each node on the global feature and further select representative nodes. The calculation formula of the global average feature is as follows:

[0081]

[0082] Where N is the number of nodes, and the sigmoid function and the hyperbolic tangent function tanh(·) are further used to calculate the attention score S a :

[0083] S a =Sigmoid(n i ·tanh(f avg ))

[0084] The attention score S a Multiplying by the feature of each graph node, we can get the aggregate feature F of the entire graph node g ∈R D :

[0085] Therefore, the visual graph nodes can be input into the above-mentioned graph convolutional neural network to obtain the corresponding visual graph node aggregation features.

[0086] In addition, the on-board terminal can traverse the constructed offline point cloud map node library, input each frame of point cloud map node data into the graph convolutional neural network, and obtain the corresponding point cloud map node aggregation features.

[0087] In one example, the traversal process can be adjusted according to the different devices used. Specifically, for CPU devices, loop traversal can be performed in a single-threaded or multi-threaded manner; for devices equipped with a GPU, batch processing can be used to set the Batch_size to achieve simultaneous processing of multiple frames of data.

[0088] In step S160, the visual graph node aggregation features and the point cloud graph node aggregation features are spliced, and the spliced ​​features are input into a fully connected network so that the fully connected network outputs the matching degree corresponding to the two modalities.

[0089] In this embodiment, the vehicle terminal can match the visual map node aggregation features with the point cloud map node aggregation features corresponding to each frame of the point cloud map node in the map library. In one example, multi-threading or batch processing can be used to accelerate the processing for different devices.

[0090] In order to enhance the point cloud node aggregation feature F pg and visual graph node aggregation feature F ig The neural tensor network uses tensor networks to learn the complex relationships between features, allowing the model to capture the interactions between multiple features.

[0091] The expression of the neural tensor network is:

[0092]

[0093] Where g∈R K Represents the output of the neural tensor network, and ReLU(·) is the activation function used to perform nonlinear transformation on the output of the network. [1:k] ∈R D×D×K Represents the weight tensor in the neural tensor network, used to learn the relationship between features, V∈R K×2D is the weight vector in the network, which is used to control the contribution of each feature to the model. [·] represents the concatenation of two dimensions. b∈R K Represents the bias vector, which adjusts the prediction effect of the network. K is a hyperparameter that represents the dimension of the similarity vector.

[0094] Then, the neural tensor network fusion feature g(F pg ,F ig ) is input into the fully connected network, and the result is simplified into a binary classification task with a value between 0 and 1. That is, the closer the matching value is to 1, the greater the possibility that the laser point cloud and the visual image are in the same scene. The fully connected network uses binary cross entropy as the loss function for network training. The true value y is set to 0 or 1, and the network output is The loss function is defined as:

[0095]

[0096] The following describes an apparatus embodiment of the present application, which can be used to implement the cross-modal scene matching method for laser point clouds and visual images described in the above-mentioned embodiments of the present application. For details not disclosed in the apparatus embodiment of the present application, please refer to the embodiment of the cross-modal scene matching method for laser point clouds and visual images described in the above-mentioned embodiments of the present application.

[0097] Figure 4 A block diagram of a cross-modal scene matching device for laser point clouds and visual images according to an embodiment of the present application is shown.

[0098] Reference Figure 4 As shown, a cross-modal scene matching device for laser point clouds and visual images according to one embodiment of the present application includes:

[0099] An image acquisition module is used to obtain images to be processed taken by the target vehicle during its driving process;

[0100] a first semantic segmentation module, configured to perform semantic segmentation on the image to be processed, and determine a semantic category corresponding to each pixel in the image to be processed, so as to obtain a semantic image;

[0101] An image clustering module, used to cluster the semantic images to obtain corresponding visual graph nodes;

[0102] A map acquisition module is used to acquire a pre-built offline point cloud map node map library, wherein the offline point cloud map node map library includes several frames of point cloud map node data;

[0103] A feature extraction module is used to extract the visual graph node aggregation features corresponding to the visual graph nodes and the point cloud graph node aggregation features corresponding to the point cloud graph node data of each frame based on a graph convolutional neural network;

[0104] The first processing module is used to splice the visual graph node aggregation features and the point cloud graph node aggregation features, and input the spliced ​​features into a fully connected network so that the fully connected network outputs the matching degree corresponding to the two modalities.

[0105] In one embodiment of the present application, the apparatus further comprises:

[0106] Point cloud acquisition module, used to obtain original point cloud data for the target road section;

[0107] A point cloud semantic segmentation module is used to perform semantic segmentation on the original point cloud data to determine the semantic category corresponding to each point in the original point cloud data and obtain a semantic point cloud;

[0108] a point cloud clustering module, configured to cluster the points based on the semantic category corresponding to each point in the semantic point cloud and the distance between the points, determine the point cloud instances in the semantic point cloud, and determine the point cloud graph nodes corresponding to each point cloud instance;

[0109] The second processing module is used to build an offline point cloud node map library based on the point cloud node data of each frame.

[0110] Figure 5 A schematic diagram of the structure of a computer system suitable for implementing an electronic device according to an embodiment of the present application is shown.

[0111] It should be noted that Figure 5 The computer system of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0112] like Figure 5 As shown, the computer system includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 502 or the program loaded from the storage part 508 to the random access memory (RAM) 503, such as executing the method described in the above embodiment. Various programs and data required for system operation are also stored in the RAM 503. The CPU 501, ROM 502 and RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0113] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, and the like; an output section 507 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 508 including a hard disk; and a communication section 509 including a network interface card such as a LAN (Local Area Network) card or a modem. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. Removable media 511, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 510 as needed, so that computer programs read therefrom can be installed into the storage section 508 as needed.

[0114] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 509, and / or installed from a removable medium 511. When the computer program is executed by the central processing unit (CPU) 501, the various functions defined in the system of the present application are executed.

[0115] It should be noted that the computer-readable medium shown in the embodiments of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable computer program. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. A computer program embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0116] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. Among them, each box in the flowchart or block diagram can represent a module, program segment, or part of the code, and the above-mentioned module, program segment, or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0117] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. In some cases, the names of these units do not constitute limitations on the units themselves.

[0118] As another aspect, the present application further provides a computer-readable medium, which may be included in the electronic device described in the above embodiments, or may exist independently without being incorporated into the electronic device. The computer-readable medium carries one or more programs, and when the one or more programs are executed by the electronic device, the electronic device implements the method described in the above embodiments.

[0119] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the application, the features and functions of two or more modules or units described above can be concretized in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0120] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the embodiments of the present application.

[0121] Those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of this application and include common knowledge or customary techniques in the art that are not disclosed herein.

[0122] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A cross-modal scene matching method for laser point clouds and visual images, characterized in that: include: Acquire the image to be processed that is taken while the target vehicle is moving; Performing semantic segmentation on the image to be processed to determine the semantic category corresponding to each pixel in the image to be processed to obtain a semantic image; Clustering the semantic image to obtain corresponding visual graph nodes; Obtain a pre-built offline point cloud map node library, wherein the offline point cloud map node library includes a plurality of frames of point cloud map node data; Based on the graph convolutional neural network, the visual graph node aggregation features corresponding to the visual graph nodes and the point cloud graph node aggregation features corresponding to the point cloud graph node data of each frame are respectively extracted; Concatenating the visual graph node aggregate features and the point cloud graph node aggregate features, and inputting the concatenated features into a fully connected network so that the fully connected network outputs the matching degree corresponding to the two modalities; Among them, building an offline point cloud map node library includes: Obtaining original point cloud data for the target road section; Performing semantic segmentation on the original point cloud data to determine the semantic category corresponding to each point in the original point cloud data to obtain a semantic point cloud; Clustering is performed based on the semantic category corresponding to each point in the semantic point cloud and the distance between points, determining point cloud instances in the semantic point cloud, and determining the point cloud graph nodes corresponding to each point cloud instance; Build an offline point cloud node map library based on the point cloud node data of each frame; The method includes clustering based on the semantic category corresponding to each point in the semantic point cloud and the distance between points, determining the point cloud instances in the semantic point cloud, and determining the point cloud graph nodes corresponding to each point cloud instance, including: According to the semantic category corresponding to each point in the semantic point cloud, the points in the semantic point cloud are divided into point sets corresponding to each semantic category; For each point in the point set, search for points in the point set within its corresponding neighborhood. If the number of points found is greater than or equal to a predetermined threshold, add the points in the neighborhood to a search queue and continue searching for the points in the search queue. Divide the searched points into the same point cloud instance, and determine the centroid information corresponding to the point cloud instance based on the coordinate information of the points in the point cloud instance; The semantic category and centroid information corresponding to the point cloud instance are stored as the point cloud graph node corresponding to the point cloud instance; The step of clustering the semantic image to obtain corresponding visual graph nodes includes: Determining depth information of each pixel in the semantic image; According to the depth information and the semantic category corresponding to each pixel, connected pixels with the same semantic category are divided into the same visual instance; The mean value of the pixel coordinate values, the average depth value and the semantic category in each of the visual instances are stored as the corresponding visual graph node; The step of combining the visual graph node aggregation features and the point cloud graph node aggregation features includes: A neural tensor network is used to splice and fuse the visual graph node aggregation features with each point cloud graph node aggregation feature.

2. The method according to claim 1, characterized in that The search neighborhood radius for each point is determined according to the following formula: Among them, α and r are adjustment coefficients related to point cloud density and semantic category, respectively, d max is the maximum distance between points in the point set, and d is the distance between points p i The Euclidean distance from the origin.

3. A cross-modal scene matching device for laser point clouds and visual images, characterized in that: include: An image acquisition module is used to obtain images to be processed taken by the target vehicle during its driving process; a first semantic segmentation module, configured to perform semantic segmentation on the image to be processed, and determine a semantic category corresponding to each pixel in the image to be processed, so as to obtain a semantic image; An image clustering module, used to cluster the semantic images to obtain corresponding visual graph nodes; A map acquisition module is used to acquire a pre-built offline point cloud map node map library, wherein the offline point cloud map node map library includes several frames of point cloud map node data; A feature extraction module is used to extract the visual graph node aggregation features corresponding to the visual graph nodes and the point cloud graph node aggregation features corresponding to the point cloud graph node data of each frame based on a graph convolutional neural network; A first processing module is configured to concatenate the visual graph node aggregate features and the point cloud graph node aggregate features, and input the concatenated features into a fully connected network so that the fully connected network outputs a matching degree corresponding to the two modalities; Wherein, the device further includes: Point cloud acquisition module, used to obtain original point cloud data for the target road section; A point cloud semantic segmentation module is used to perform semantic segmentation on the original point cloud data to determine the semantic category corresponding to each point in the original point cloud data and obtain a semantic point cloud; a point cloud clustering module, configured to cluster the points based on the semantic category corresponding to each point in the semantic point cloud and the distance between the points, determine the point cloud instances in the semantic point cloud, and determine the point cloud graph nodes corresponding to each point cloud instance; The second processing module is used to build an offline point cloud node map library based on the point cloud node data of each frame; The point cloud clustering module is further configured to divide the points in the semantic point cloud into point sets corresponding to the semantic categories according to the semantic categories corresponding to each point in the semantic point cloud; For each point in the point set, search for points in the point set within its corresponding neighborhood. If the number of points found is greater than or equal to a predetermined threshold, add the points in the neighborhood to a search queue and continue searching for the points in the search queue. Divide the searched points into the same point cloud instance, and determine the centroid information corresponding to the point cloud instance based on the coordinate information of the points in the point cloud instance; The semantic category and centroid information corresponding to the point cloud instance are stored as the point cloud graph node corresponding to the point cloud instance; The image clustering module is further configured to determine depth information of each pixel in the semantic image; According to the depth information and the semantic category corresponding to each pixel, connected pixels with the same semantic category are divided into the same visual instance; The mean value of the pixel coordinate values, the average depth value and the semantic category in each of the visual instances are stored as the corresponding visual graph node; The step of combining the visual graph node aggregation features and the point cloud graph node aggregation features includes: A neural tensor network is used to splice and fuse the visual graph node aggregation features with each point cloud graph node aggregation feature.

4. A computer-readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the cross-modal scene matching method of laser point cloud and visual image according to any one of claims 1 to 2 is implemented.

5. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the cross-modal scene matching method of laser point cloud and visual image as described in any one of claims 1 to 2.

Images