A classification method and device for integrating image point cloud features

Abstract

Embodiments of the present invention relate to a method and apparatus for classifying fused image point cloud features. The method comprises: obtaining a first image sequence from multiple images captured by a specified camera within a specified time period, and generating a first point cloud sequence from multiple scanned point clouds generated by a specified radar scan; performing image point cloud fusion feature extraction processing based on the first image sequence and the first point cloud sequence to generate a corresponding fused feature vector; classifying the fused feature vector using a classification neural network to obtain a corresponding classification vector; and outputting the classification category corresponding to the classification vector data with the largest value as the classification result. This invention can improve classification accuracy.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a classification method and device for fusing image point cloud features. Background Art

[0002] The perception module of an autonomous driving system classifies obstacles in the vehicle's driving environment based on sensor data (camera images, radar point clouds) acquired by sensors (cameras, radars, etc.) and transmits the classification results to downstream modules. During classification, the perception module typically performs classification based on image features and point cloud features, then compares the two classification results. If the classifications match or overlap, the intersecting parts are output as the classification result. This approach requires executing two complete classification processes, which is time-consuming. Furthermore, the range of the classification intersection is difficult to determine, which can lead to excessively large classification ranges. Summary of the Invention

[0003] The purpose of the present invention is to address the shortcomings of the existing technology and provide a classification method, device, electronic device and computer-readable storage medium for fusing image point cloud features, select images and point clouds in the most recent period to form corresponding image and point cloud sequences, and perform feature extraction on the image sequence based on the ResNet18 neural network, and on the point cloud sequence based on the PointNet neural network, perform feature merging on the extracted image and point cloud features, and perform feature fusion on the merged vector based on the Long Short-Term Memory (LSTM) network, and then use a classification neural network to perform classification processing based on the feature fusion vector. Through the present invention, on the one hand, only two sets of feature processing processes need to be executed in parallel, which shortens the processing time; on the other hand, classification is performed based on the fusion features of the image and point cloud, and the classification results are clearer and the classification accuracy is higher.

[0004] To achieve the above-mentioned object, a first aspect of an embodiment of the present invention provides a classification method for fusing image point cloud features, the method comprising:

[0005] Acquire multiple images captured by a specified camera within a specified time period to generate a first image sequence; and acquire multiple scan point clouds generated by a specified radar scan corresponding to the specified camera within the specified time period to generate a first point cloud sequence; the first image sequence includes multiple first images, and the first point cloud sequence includes multiple first point clouds;

[0006] Performing image point cloud fusion feature extraction processing on the first image sequence and the first point cloud sequence to generate a corresponding fusion feature vector;

[0007] The fused feature vector is classified based on a classification neural network to obtain a corresponding classification vector; the classification vector is a one-dimensional vector, including a specified number m of classification vector data, each of which corresponds to a classification category; the classification categories include six categories, namely, vehicles, pedestrians, cyclists, green plants, fences, and other objects;

[0008] The classification category corresponding to the classification vector data with the largest value is output as the classification result of this time.

[0009] Preferably, performing image point cloud fusion feature extraction processing on the first image sequence and the first point cloud sequence to generate a corresponding fusion feature vector specifically includes:

[0010] Performing image feature extraction processing on the first image sequence to generate a corresponding first feature vector;

[0011] Performing point cloud feature extraction processing on the first point cloud sequence to generate a corresponding second feature vector;

[0012] Performing vector merging on the first and second eigenvectors to obtain a corresponding third eigenvector;

[0013] The third eigenvector is input into the LSTM neural network for feature fusion processing to generate a corresponding fused feature vector.

[0014] Furthermore, the performing image feature extraction processing according to the first image sequence to generate a corresponding first feature vector specifically includes:

[0015] Counting the number of the first images in the first image sequence to generate a first number b1;

[0016] Input each of the first images in the first image sequence into the ResNet18 neural network for feature extraction to generate a corresponding first image feature vector; the shape of the first image feature vector is 1×1024;

[0017] The first feature vector is composed of the obtained first number b1 of the first image feature vectors; the shape of the first feature vector is b1×1024.

[0018] Furthermore, performing point cloud feature extraction processing according to the first point cloud sequence to generate a corresponding second feature vector specifically includes:

[0019] Counting the number of the first point clouds in the first point cloud sequence to generate a second number b2;

[0020] Input each of the first point clouds in the first point cloud sequence into the PointNet neural network for feature extraction to generate a corresponding first point cloud feature vector; the shape of the first point cloud feature vector is 1×1024;

[0021] The second feature vector is composed of the obtained second number b2 of the first point cloud feature vectors; the shape of the second feature vector is b2×1024.

[0022] Furthermore, the shape of the third eigenvector is (b1+b2)×1024; the first number b1 is the number of the first images in the first image sequence, and the second number b2 is the number of the first point clouds in the first point cloud sequence.

[0023] Furthermore, the step of inputting the third feature vector into the LSTM neural network for feature fusion processing to generate a corresponding fused feature vector specifically includes:

[0024] The third eigenvector with a structure of (b1+b2)×1024 is input into the LSTM neural network, and the LSTM neural network performs feature fusion on the (b1+b2) features on the 1024 dimensions of the third eigenvector to obtain the corresponding fused feature vector; the structure of the fused feature vector is 1×1024.

[0025] Preferably, the classification neural network comprises a fully connected network layer and an activation network layer; the classification processing of the fused feature vector based on the classification neural network to obtain the corresponding classification vector specifically includes:

[0026] The fused feature vector is input into the fully connected network layer of the classification neural network for full connection operation to output a corresponding fully connected vector; and the fully connected vector is input into the activation network layer for classification scoring to obtain the corresponding classification vector.

[0027] A second aspect of the embodiments of the present invention provides a device for implementing the classification method of fused image point cloud features described in the first aspect, the device comprising: an acquisition module, a feature fusion module, and a classification module;

[0028] The acquisition module is configured to acquire a plurality of images captured by a specified camera within a specified time period to generate a first image sequence; and acquire a plurality of scan point clouds generated by a specified radar scan corresponding to the specified camera within the specified time period to generate a first point cloud sequence; the first image sequence includes a plurality of first images, and the first point cloud sequence includes a plurality of first point clouds;

[0029] The feature fusion module is used to perform image point cloud fusion feature extraction processing based on the first image sequence and the first point cloud sequence to generate a corresponding fusion feature vector;

[0030] The classification module is used to classify the fused feature vector based on a classification neural network to obtain a corresponding classification vector; and output the classification category corresponding to the classification vector data with the largest value as the classification result of this classification; the classification vector is a one-dimensional vector, including a specified number m of classification vector data, and each of the classification vector data corresponds to a classification category; the classification categories include 6 categories, namely vehicles, pedestrians, cyclists, green plants, fences and other objects.

[0031] A third aspect of an embodiment of the present invention provides an electronic device, including: a memory, a processor, and a transceiver;

[0032] The processor is configured to be coupled to the memory, read and execute instructions in the memory, so as to implement the method steps described in the first aspect above;

[0033] The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

[0034] A fourth aspect of an embodiment of the present invention provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed by a computer, the computer executes the instructions of the method described in the first aspect above.

[0035] The embodiments of the present invention provide a classification method, device, electronic device, and computer-readable storage medium for fusing image point cloud features. The method selects images and point clouds from the most recent period to form corresponding image and point cloud sequences, extracts features from the image sequence based on the ResNet18 neural network, extracts features from the point cloud sequence based on the PointNet neural network, merges the extracted image and point cloud features into vectors, fuses the merged vectors based on LSTM, and then uses a classification neural network to perform classification processing based on the feature fusion vectors. The present invention reduces processing time by only having to execute two sets of feature processing processes in parallel. Furthermore, classification based on the fused features of images and point clouds makes the classification results clearer and improves classification accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of a classification method for fusing image point cloud features provided in the first embodiment of the present invention;

[0037] Figure 2 A module structure diagram of a classification device for fusing image point cloud features provided in the second embodiment of the present invention;

[0038] Figure 3 This is a structural diagram of an electronic device provided in Example 3 of the present invention. DETAILED DESCRIPTION

[0039] To make the objectives, technical solutions, and advantages of the present invention more apparent, the present invention will be further described in detail below with reference to the accompanying drawings. It should be understood that the embodiments described herein are merely some, rather than all, of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are intended to fall within the scope of protection of the present invention.

[0040] The first embodiment of the present invention provides a classification method for integrating image point cloud features, such as Figure 1 A schematic diagram of a classification method for integrating image point cloud features provided in the first embodiment of the present invention is shown. The method mainly includes the following steps:

[0041] Step 1: Acquire multiple images captured by a specified camera within a specified time period to generate a first image sequence; and acquire multiple scan point clouds generated by a specified radar scan corresponding to the specified camera within the specified time period to generate a first point cloud sequence;

[0042] The first image sequence includes multiple first images, and the first point cloud sequence includes multiple first point clouds. The specified time period can be set arbitrarily, with the default being the most recent time period. The perception module acquires the first image sequence from a specified camera and the first point cloud sequence from a specified radar. The specified camera and the specified radar observe the same scene and at the same time. The first image sequence is the image sequence captured by the specified camera during the specified time period, and the first point cloud sequence is the point cloud sequence scanned by the specified radar during the specified time period.

[0043] Step 2: performing image and point cloud fusion feature extraction processing on the first image sequence and the first point cloud sequence to generate a corresponding fusion feature vector;

[0044] Here, the embodiment of the present invention first extracts features from the first image sequence and the first point cloud sequence respectively, and then merges and fuses the feature vectors of the two to obtain a fused feature vector;

[0045] Specifically comprising: step 21, performing image feature extraction processing according to the first image sequence to generate a corresponding first feature vector;

[0046] Specifically, it includes: step 211, counting the number of first images in the first image sequence to generate a first number b1;

[0047] Step 212: Input each first image in the first image sequence into the ResNet18 neural network for feature extraction to generate a corresponding first image feature vector;

[0048] Among them, the shape of the first image feature vector is 1×1024;

[0049] Here, from the paper "Deep Residual Learning for Image Recognition" published by Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun from Microsoft Research, it can be seen that the network structure of the ResNet18 neural network consists of 1 convolutional layer, 4 residual units ResBlock (each ResBlock consists of conv1 layer, conv2_x layer, conv3_x layer, conv4_x layer, conv5_x layer and the last classifier; among them, the conv1 layer is a traditional convolutional network layer, the conv2_x layer, conv3_x layer, conv4_x layer, and conv5_x layer are residual network layers, each residual network layer includes 2 continuous residual modules, each continuous residual module includes 2 convolutional layers, the classifier consists of a pooling layer, a fully connected layer, and a softmax classification function. The total number of network layers of the ResNet18 neural network is 1+4*2*2+1=18 layers;

[0050] It should be noted that, when the embodiment of the present invention inputs each first image of the first image sequence into the ResNet18 neural network for feature extraction processing, only the conv1 layer, conv2_x layer, conv3_x layer, conv4_x layer, and conv5_x layer of the ResNet18 neural network are used for calculation, and the classifier of the network is not used for classification; in addition, the number of data channels of the output feature vector of the conv5_x layer is less than 1024, and the number of data channels of the output feature vector needs to be expanded to 1024 by padding; therefore, each first image of the first image sequence is input into the ResNet18 neural network for feature extraction processing to generate the corresponding first The image feature vector is specifically as follows: the current first image is input into the conv1 layer of the ResNet18 neural network for calculation to obtain the corresponding first output vector, and the first output vector is input into the conv2_x layer for calculation to obtain the corresponding second output vector, and the second output vector is input into the conv3_x layer for calculation to obtain the corresponding third output vector, and the third output vector is input into the conv4_x layer for calculation to obtain the corresponding fourth output vector, and the fourth output vector is input into the conv5_x layer for calculation to obtain the corresponding fifth output vector, and the data channel of the fifth output vector is expanded to 1024 by padding with 0 to obtain a first image feature vector with a shape of 1×1024;

[0051] Step 213, forming a first feature vector from the obtained first number b1 first image feature vectors;

[0052] Among them, the shape of the first eigenvector is b1×1024;

[0053] Step 22: performing point cloud feature extraction processing on the first point cloud sequence to generate a corresponding second feature vector;

[0054] Specifically, it includes: step 221, counting the number of first point clouds in the first point cloud sequence to generate a second number b2;

[0055] Step 222: Input each first point cloud in the first point cloud sequence into the PointNet neural network for feature extraction to generate a corresponding first point cloud feature vector;

[0056] Among them, the shape of the first point cloud feature vector is 1×1024;

[0057] Here, the paper "PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation" published by authors Charles R. Qi*, Hao Su*, Kaichun Mo, and Leonidas J. Guibas of Stanford University shows that the PointNet neural network has two application network structures: a classification network structure and a segmentation network structure. The embodiment of the present invention uses the classification network to extract features from the point cloud branch sequence; the classification network structure is composed of a multi-level feature conversion network, a maximum pooling network, and a fully connected classification network; wherein the multi-level feature conversion network is used to perform multi-level feature conversion on the input point cloud, the maximum pooling network is used to perform feature aggregation on the output vector of the multi-level feature conversion network, and the fully connected classification network obtains the final classification vector by performing three-level full connection on the output vector of the maximum pooling network, where the number of data channels of the maximum pooling network output vector is 1024;

[0058] It should be noted that, in the embodiment of the present invention, when each first point cloud of the first point cloud sequence is input into the PointNet neural network for feature extraction processing, only the multi-level feature conversion network and the maximum pooling network of the classification network of the PointNet neural network are used for calculation; therefore, each first point cloud of the first point cloud sequence is input into the PointNet neural network for feature extraction processing to generate the corresponding first point cloud feature vector specifically as follows: the current first point cloud is input into the multi-level feature conversion network for calculation to obtain the corresponding first output vector, and the first output vector is input into the maximum pooling network for calculation to obtain the first point cloud feature vector with a shape of 1×1024;

[0059] Step 223, forming a second feature vector from the obtained second number b2 of first point cloud feature vectors;

[0060] Among them, the shape of the second eigenvector is b2×1024;

[0061] Step 23, performing vector merging on the first and second eigenvectors to obtain a corresponding third eigenvector;

[0062] Among them, the shape of the third eigenvector is (b1+b2)×1024;

[0063] Step 24: Input the third feature vector into the LSTM neural network for feature fusion processing to generate a corresponding fused feature vector;

[0064] Specifically, the third eigenvector with a structure of (b1+b2)×1024 is input into the LSTM neural network, and the LSTM neural network performs feature fusion on the (b1+b2) features on the 1024 dimensions of the third eigenvector to obtain the corresponding fused feature vector; the structure of the fused feature vector is 1×1024.

[0065] Here, from the article "Long short-term memory" published by the authors Hochreiter, S. and J. Schmidhuber, we can know the network structure of the LSTM neural network. LSTM is a type of recurrent neural network (RNN). Its network operation process can be found in the content of the article, so I will not go into details here.

[0066] Step 3: Classify the fused feature vector based on the classification neural network to obtain the corresponding classification vector;

[0067] The classification neural network consists of a fully connected network layer and an activation network layer. The classification vector is a one-dimensional vector, including a specified number m of classification vector data, each of which corresponds to a classification category. The classification categories include six categories: cars, pedestrians, cyclists, green plants, fences, and other objects.

[0068] Specifically, the method includes: inputting the fused feature vector into the fully connected network layer of the classification neural network to perform a fully connected operation to output the corresponding fully connected vector; and inputting the fully connected vector into the activation network layer to perform classification scoring to obtain the corresponding classification vector.

[0069] Here, the specified number m defaults to 6, and the resulting classification vector includes 6 classification vector data, each of which corresponds to one of the 6 categories (cars, pedestrians, cyclists, green plants, fences, and other objects). Each classification vector data is actually the classification score (or classification probability) of the corresponding category.

[0070] Step 4: Output the classification category corresponding to the classification vector data with the largest value as the classification result.

[0071] Here, the classification vector data with the largest value is selected from the 6 classification vector data of the classification vector as the maximum classification score, the classification category corresponding to the maximum classification score is used as the maximum score category, and finally the maximum score category is output as the classification result of this fusion image point cloud feature.

[0072] Figure 2This is a module structure diagram of a classification device for integrating image point cloud features provided in the second embodiment of the present invention. The device is a terminal device or server that implements the aforementioned method embodiment, or can be a device that enables the aforementioned terminal device or server to implement the aforementioned method embodiment. For example, the device can be a device or chip system of the aforementioned terminal device or server. Figure 2 As shown, the device includes: an acquisition module 201, a feature fusion module 202, and a classification module 203.

[0073] The acquisition module 201 is used to acquire multiple images captured by a specified camera within a specified time period to generate a first image sequence; and acquire multiple scanning point clouds generated by a specified radar scan corresponding to the specified camera within the specified time period to generate a first point cloud sequence; the first image sequence includes multiple first images, and the first point cloud sequence includes multiple first point clouds.

[0074] The feature fusion module 202 is used to perform image point cloud fusion feature extraction processing according to the first image sequence and the first point cloud sequence to generate a corresponding fusion feature vector.

[0075] The classification module 203 is used to classify the fused feature vector based on the classification neural network to obtain the corresponding classification vector; and output the classification category corresponding to the classification vector data with the largest value as the classification result of this time; the classification vector is a one-dimensional vector, including a specified number m of classification vector data, and each classification vector data corresponds to a classification category; the classification categories include 6 categories, namely cars, pedestrians, cyclists, green plants, fences and other objects.

[0076] An embodiment of the present invention provides a classification device for fusing image point cloud features, which can execute the method steps in the above method embodiment. Its implementation principles and technical effects are similar and will not be repeated here.

[0077] It should be noted that it should be understood that the division of the various modules of the above device is merely a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. Moreover, these modules can all be implemented in the form of software called by a processing element; or they can all be implemented in the form of hardware; or some modules can be implemented in the form of software called by a processing element, and some modules can be implemented in the form of hardware. For example, the acquisition module can be a separately established processing element, or it can be integrated into a chip of the above device. In addition, it can also be stored in the memory of the above device in the form of program code, and called by a processing element of the above device to perform the functions of the above determination module. The implementation of other modules is similar. In addition, these modules can all or partly be integrated together, or they can be implemented independently. The processing element described here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed by the hardware integrated logic circuit in the processor element or by instructions in the form of software.

[0078] For example, the above modules may be one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs). For another example, when a module is implemented by scheduling program code through a processing element, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call program code. For another example, these modules may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0079] In the above embodiments, all or part of the embodiments may be implemented using software, hardware, firmware, or any combination thereof. When implemented using software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the above method embodiments are generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The above-mentioned computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the above-mentioned computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, Bluetooth, microwave, etc.) means. The above-mentioned computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media. The above-mentioned available medium can be a magnetic medium (such as a floppy disk, hard disk, tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0080] Figure 3 This is a schematic diagram of the structure of an electronic device provided in the third embodiment of the present invention. The electronic device may be the aforementioned terminal device or server, or may be a terminal device or server connected to the aforementioned terminal device or server to implement the method of the embodiment of the present invention. Figure 3 As shown, the electronic device may include: a processor 301 (such as a CPU), a memory 302, and a transceiver 303; the transceiver 303 is coupled to the processor 301, and the processor 301 controls the transceiver 303's transceiver actions. Various instructions may be stored in the memory 302 for completing various processing functions and implementing the processing steps described in the aforementioned method embodiment. Preferably, the electronic device involved in the embodiment of the present invention further includes: a power supply 304, a system bus 305, and a communication port 306. The system bus 305 is used to realize communication connections between components. The above-mentioned communication port 306 is used for connection and communication between the electronic device and other peripherals.

[0081] exist Figure 3The system bus 305 mentioned in the figure can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The system bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 The use of a single bold line in the diagram does not necessarily imply a single bus or type of bus. Communication interfaces enable communication between the database access device and other devices (e.g., clients, read-write libraries, and read-only libraries). Memory may include random access memory (RAM) and non-volatile memory (NVM), such as at least one disk drive.

[0082] The above-mentioned processors can be general-purpose processors, including central processing units (CPUs), network processors (NPs), graphics processing units (GPUs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0083] It should be noted that an embodiment of the present invention further provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is run on a computer, it enables the computer to execute the methods and processing procedures provided in the above embodiments.

[0084] An embodiment of the present invention further provides a chip for executing instructions, which is used to execute the processing steps described in the above method embodiment.

[0085] The embodiments of the present invention provide a classification method, device, electronic device, and computer-readable storage medium for fusing image point cloud features. The method selects images and point clouds from the most recent period to form corresponding image and point cloud sequences, extracts features from the image sequence based on the ResNet18 neural network, extracts features from the point cloud sequence based on the PointNet neural network, merges the extracted image and point cloud features into vectors, fuses the merged vectors based on LSTM, and then uses a classification neural network to perform classification processing based on the feature fusion vectors. The present invention reduces processing time by only having to execute two sets of feature processing processes in parallel. Furthermore, classification based on the fused features of images and point clouds makes the classification results clearer and improves classification accuracy.

[0086] Professionals should also be further aware that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0087] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0088] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A classification method for integrating image point cloud features, characterized in that: The method comprises: Acquire multiple images captured by a specified camera within a specified time period to generate a first image sequence; and acquire multiple scan point clouds generated by a specified radar scan corresponding to the specified camera within the specified time period to generate a first point cloud sequence; the first image sequence includes multiple first images, and the first point cloud sequence includes multiple first point clouds; Performing image point cloud fusion feature extraction processing on the first image sequence and the first point cloud sequence to generate a corresponding fusion feature vector; The fused feature vector is classified based on a classification neural network to obtain a corresponding classification vector; the classification vector is a one-dimensional vector, including a specified number m of classification vector data, each of which corresponds to a classification category; the classification categories include six categories, namely, vehicles, pedestrians, cyclists, green plants, fences, and other objects; Output the classification category corresponding to the classification vector data with the largest value as the classification result of this time; The designated time period is a recent time period; the designated camera and the designated radar observe the same scene and at the same time; the first image sequence is an image sequence captured by the designated camera during the designated time period; and the first point cloud sequence is a point cloud sequence scanned by the designated radar during the designated time period. The performing image-point cloud fusion feature extraction processing based on the first image sequence and the first point cloud sequence to generate a corresponding fusion feature vector specifically includes: performing image feature extraction processing based on the first image sequence to generate a corresponding first feature vector; performing point cloud feature extraction processing based on the first point cloud sequence to generate a corresponding second feature vector; performing vector merging on the first and second feature vectors to obtain a corresponding third feature vector; and inputting the third feature vector into an LSTM neural network to perform feature fusion processing to generate a corresponding fusion feature vector; The performing image feature extraction processing based on the first image sequence to generate a corresponding first feature vector specifically includes: counting the number of the first images in the first image sequence to generate a first number b1; inputting each of the first images in the first image sequence into a ResNet18 neural network to perform feature extraction processing to generate a corresponding first image feature vector; the shape of the first image feature vector is 1×1024; and forming the first feature vector from the obtained first number b1 of the first image feature vectors; the shape of the first feature vector is b1×1024; The step of performing point cloud feature extraction processing on the first point cloud sequence to generate a corresponding second feature vector specifically includes: performing statistical analysis on the number of the first point clouds in the first point cloud sequence to generate a second number b2; inputting each of the first point clouds in the first point cloud sequence into a PointNet neural network for feature extraction processing to generate a corresponding first point cloud feature vector; the shape of the first point cloud feature vector is 1×1024; and forming a second feature vector from the obtained second number b2 of the first point cloud feature vectors; the shape of the second feature vector is b2×1024; The shape of the third eigenvector is (b1+b2)×1024; the first number b1 is the number of the first images in the first image sequence, and the second number b2 is the number of the first point clouds in the first point cloud sequence; Inputting the third eigenvector into the LSTM neural network for feature fusion processing to generate a corresponding fused feature vector specifically includes: inputting the third eigenvector having a structure of (b1+b2)×1024 into the LSTM neural network, and having the LSTM neural network perform feature fusion on the (b1+b2) features on 1024 dimensions of the third eigenvector, thereby obtaining the corresponding fused feature vector; the structure of the fused feature vector is 1×1024; The classification neural network includes a fully connected network layer and an activation network layer; The classification processing of the fused feature vector based on the classification neural network to obtain the corresponding classification vector specifically includes: inputting the fused feature vector into the fully connected network layer of the classification neural network to perform a fully connected operation to output the corresponding fully connected vector; and inputting the fully connected vector into the activation network layer to perform classification scoring to obtain the corresponding classification vector.

2. A device for implementing the classification method of fused image point cloud features according to claim 1, characterized in that: The device includes: an acquisition module, a feature fusion module, and a classification module; The acquisition module is configured to acquire a plurality of images captured by a specified camera within a specified time period to generate a first image sequence; and acquire a plurality of scan point clouds generated by a specified radar scan corresponding to the specified camera within the specified time period to generate a first point cloud sequence; the first image sequence includes a plurality of first images, and the first point cloud sequence includes a plurality of first point clouds; The feature fusion module is used to perform image point cloud fusion feature extraction processing based on the first image sequence and the first point cloud sequence to generate a corresponding fusion feature vector; The classification module is used to classify the fused feature vector based on a classification neural network to obtain a corresponding classification vector; and output the classification category corresponding to the classification vector data with the largest value as the classification result of this classification; the classification vector is a one-dimensional vector, including a specified number m of classification vector data, and each of the classification vector data corresponds to a classification category; the classification categories include 6 categories, namely vehicles, pedestrians, cyclists, green plants, fences and other objects.

3. An electronic device, characterized in that: include: memory, processors, and transceivers; The processor is configured to be coupled to the memory, read and execute instructions in the memory, so as to implement the method according to claim 1; The transceiver is coupled to the processor, and the processor controls the transceiver to send and receive messages.

4. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, which, when executed by a computer, cause the computer to perform the method of claim 1 .

Images