Classification method and device based on multi-view continuous learning, equipment and storage medium

Abstract

The application relates to the technical field of vehicles, in particular to a classification method and device based on multi-view continuous learning, equipment and a storage medium, wherein the method comprises the following steps: acquiring first-view image data and second-view image data, fusing the first-view image data and the second-view image data to obtain double-view fusion data, inputting the double-view fusion data into a plurality of pre-learned classifiers, obtaining the probability of the category to which the double-view fusion data output by each classifier belongs, and aggregating the probability of the category to which the double-view fusion data output by each classifier belongs to obtain a final classification result. Therefore, the fusion process of the traditional deep learning model has certain contingency, the model cannot remember or recognize the previous state, and the problems that it is difficult to handle the re-access to the previous state are solved, and the safety and stability of the unmanned driving system in a complex scene are further improved.

Description

Technical Field

[0001] The present application relates to the field of vehicle technology, and in particular to a classification method, apparatus, device and storage medium based on multi-perspective continuous learning. Background Art

[0002] With the development of intelligent driving technology, autonomous driving systems are acquiring more diverse data and information. The amount of data that can be collected is increasing, the data structure is becoming more complex, and the data dimension is increasing. Image fusion involves combining image data of the same target collected from multiple channels through image processing and computer technology to maximize the extraction of useful information from each channel. The final fusion is then combined into a high-quality image to reduce the uncertainty of the scene description in the image. Image fusion can effectively improve the utilization rate of the original image information.

[0003] In the related art, in the existing technology related to image fusion based on deep learning, an image fusion method based on convolutional neural network is usually used to perform image fusion processing.

[0004] However, the image fusion method based on deep learning in related technologies can only perform image fusion for specific scenes or specific multispectral situations, which leads to a limited application scope of image fusion and urgently needs to be improved. Summary of the Invention

[0005] The present application provides a classification method, apparatus, device and storage medium based on multi-perspective continuous learning to solve the problems that the fusion process of traditional deep learning models has a certain degree of randomness, the model cannot remember or recognize the previous state, and it is difficult to handle the revisit of the previous state, thereby further improving the safety and stability of unmanned driving systems in complex scenarios.

[0006] The first embodiment of the present application provides a classification method based on multi-view continuous learning, comprising the following steps:

[0007] Acquire first-perspective image data and second-perspective image data;

[0008] fusing the first-view image data and the second-view image data to obtain dual-view fusion data, and inputting the dual-view fusion data into a plurality of pre-learned classifiers to obtain a probability of the category to which the dual-view fusion data belongs, output by each classifier; and

[0009] Based on the probability of the category to which the dual-view fusion data output by each classifier belongs, the final classification result is obtained by aggregation.

[0010] Based on the above technical means, this application designs a coupled autoencoder and integration model consisting of a pair of pre-trained autoencoders and a set of simple single-layer classifiers to fuse data from different perspectives of multiple sensors, thereby further improving the safety and stability of unmanned driving systems in complex scenarios.

[0011] Furthermore, before inputting the dual-view fusion data into the pre-learned multiple classifiers, the method further includes:

[0012] Acquire a data set to be trained, wherein the data set to be trained includes first-perspective training data and second-perspective training data;

[0013] Using a pre-trained autoencoder to fuse the first-perspective training data and the second-perspective training data to obtain a fused implicit representation;

[0014] Based on a preset minimization loss function, the fused implicit representation is used to train an initial classification model to obtain the pre-learned multiple classifiers, wherein the initial classification model is composed of multiple single-layer classifiers and multi-view image data.

[0015] Based on the above technical means, this application establishes a unified and effective framework for selecting classifiers through data fusion to further improve reliability.

[0016] Furthermore, obtaining the dataset to be trained includes:

[0017] Obtain the Noisy-MNIST and Half-MNIST perspective image datasets;

[0018] According to a preset segmentation strategy, the Noisy-MNIST perspective image dataset and the Half-MNIST perspective image dataset are both segmented into the to-be-trained dataset consisting of a preset number of images.

[0019] According to the above technical means, the present application captures data information from different perspectives and divides the data sets from different perspectives according to a preset segmentation strategy to obtain the data set to be trained, thereby improving the rigor of the test results.

[0020] Furthermore, the preset minimization loss function is:

[0021]

[0022] in, are the training samples of the first and second perspectives respectively, y t is the label corresponding to the sample, Θ=[θ1,θ2,...θ n ] represents the parameters of the N single-layer classifiers that constitute the continuous learning module.

[0023] Based on the above technical means, this application has more practical application prospects by performing standardized mathematical representation and modeling.

[0024] A second embodiment of the present application provides a classification device based on multi-view continuous learning, comprising:

[0025] An acquisition module, configured to acquire first-perspective image data and second-perspective image data;

[0026] a processing module, configured to fuse the first-view image data and the second-view image data to obtain dual-view fusion data, and input the dual-view fusion data into a plurality of pre-learned classifiers to obtain a probability of the category to which the dual-view fusion data belongs, output by each classifier; and

[0027] An aggregation module is used to aggregate the probabilities of the categories to which the dual-view fusion data output by each classifier belongs to obtain a final classification result.

[0028] Furthermore, before inputting the dual-view fusion data into the pre-learned multiple classifiers, the processing module is further configured to:

[0029] Acquire a data set to be trained, wherein the data set to be trained includes first-perspective training data and second-perspective training data;

[0030] Using a pre-trained autoencoder to fuse the first-perspective training data and the second-perspective training data to obtain a fused implicit representation;

[0031] Based on a preset minimization loss function, the fused implicit representation is used to train an initial classification model to obtain the pre-learned multiple classifiers, wherein the initial classification model is composed of multiple single-layer classifiers and multi-view image data.

[0032] Furthermore, the processing module is used to obtain the training data set:

[0033] Obtain the Noisy-MNIST and Half-MNIST perspective image datasets;

[0034] According to a preset segmentation strategy, the Noisy-MNIST perspective image dataset and the Half-MNIST perspective image dataset are both segmented into the to-be-trained dataset consisting of a preset number of images.

[0035] Furthermore, the preset minimization loss function is:

[0036]

[0037] in, are the training samples of the first and second perspectives respectively, y t is the label corresponding to the sample, Θ=[θ1,θ2,...θ n ] represents the parameters of the N single-layer classifiers that constitute the continuous learning module.

[0038] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the program to implement the classification method based on multi-perspective continuous learning as described in the above embodiment.

[0039] The fourth aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the classification method based on multi-perspective continuous learning as described in the above embodiment.

[0040] Therefore, the present application has at least the following beneficial effects: (1) By integrating the multi-view fusion module into the continuous learning module, the continuous learning algorithm can make full use of data from multiple perspectives and utilize their complementarity to improve the robustness of the system; (2) The present application establishes a unified and effective framework through the memory space to select classifiers, which is more reliable and not restricted by the task, and has more practical application prospects; (3) The present application quantifies the conceptual clustering effect, improving the rigor of the test results. Thus, the problem that the fusion process of the traditional deep learning model has a certain degree of randomness and the model cannot remember or recognize the previous state, resulting in difficulty in handling the revisit of the previous state is solved, thereby further improving the safety and stability of the unmanned driving system in complex scenarios.

[0041] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0043] Figure 1 A flowchart of a classification method based on multi-perspective continuous learning provided according to an embodiment of the present application;

[0044] Figure 2 is a schematic diagram of a CAEE model architecture according to one embodiment of the present application;

[0045] Figure 3Schematic diagram of latent space generated by AE model and CAE model according to one embodiment of the present application. Figure 3 (a) is the Sihouette distribution diagram of samples in each cluster, Figure 3 (b) is a schematic diagram of the latent space generated by AE. Figure 3 (c) is the Sihouette distribution diagram of samples in each cluster, Figure 3 (d) Schematic diagram of the latent space generated by CAE;

[0046] Figure 4 A schematic diagram of a Noisy-MNIST dataset sample according to one embodiment of the present application;

[0047] Figure 5 The figure is a schematic diagram comparing the accuracy of the CAEE model and the Vanilla model on Noisy-MNIST according to one embodiment of the present application. Figure 5 (a) Noisy MNIST task 1 accuracy, Figure 5 (b) Noisy MNIST task 2 accuracy, Figure 5 (c) Noisy MNIST-overall accuracy;

[0048] Figure 6 is a block diagram of a classification device based on multi-view continuous learning according to an embodiment of the present application;

[0049] Figure 7 Schematic diagram of the structure of an electronic device according to an embodiment of the present application.

[0050] Explanation of the accompanying drawings: 10 - classification device based on multi-perspective continuous learning, 100 - acquisition module, 200 - processing module, 300 - aggregation module, 701 - memory, 702 - processor, 703 - communication interface. DETAILED DESCRIPTION

[0051] The following describes in detail embodiments of the present application. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0052] The following describes the classification method, apparatus, device and storage medium based on multi-perspective continuous learning according to the embodiments of the present application with reference to the accompanying drawings.

[0053] Before introducing the classification method based on multi-view continuous learning proposed in this application, a brief introduction to the classification method based on multi-view continuous learning in related technologies is first given.

[0054] Related technology proposes an object recognition method and a robot for robots, which shoots objects from multiple angles as a database. In subsequent recognition tasks, the image of the object is compared with the image database of the corresponding angle, and the one with the highest similarity is selected as the final classification result.

[0055] However, this method does not exploit the complementarity between multiple views and only performs retrieval based on the features of a single view.

[0056] Related technologies also propose a multi-perspective enhanced image clustering method, which makes full use of the consistency and complementarity of multiple perspectives to improve the effect of image clustering. It adopts a two-step method. First, an independent feature extraction network is established for each perspective to extract the unique features of the perspective; then a multi-perspective feature fusion network is trained to fuse the features of multiple perspectives to obtain the fusion feature space of each perspective.

[0057] However, this method has two problems: (1) the features of the perspectives are extracted first, and then the relationship between the perspectives is established. This two-step method of fusing multi-perspective features will hinder their performance in cross-perspective recognition tasks, because it is difficult to ensure the integrity of the original information in two steps, and the distribution that deviates from the original information may be learned, which brings a certain degree of randomness to the fusion process and low robustness; (2) this method is difficult to learn multiple tasks in sequence, and the previously learned tasks will be forgotten when learning new tasks. When the input data or learning objectives change, the traditional deep learning model will learn and adapt to the new state, but the model will not remember or recognize the previous state, making it difficult to handle revisiting the previous state, which will lead to reduced performance.

[0058] Furthermore, for single-expert selection networks, the key to combating forgetting lies in dividing tasks and measuring the similarity between them, thereby selecting appropriate experts for learning. This allows experts to become proficient in a particular task, and subsequent selection of the corresponding expert is based solely on the characteristics of the task, effectively avoiding catastrophic forgetting. Expert Gate's approach utilizes an autoencoder to capture the characteristics of each task. During the training phase, a new model (expert) is learned for each new task. This model is initialized through knowledge transfer from the model with the closest similarity to the other task. This allows Expert Gate's autoencoders to distinguish between different tasks with remarkable accuracy. They can also be used to select the most relevant tasks and the most appropriate transfer methods during training.

[0059] However, these methods are all task-based, requiring each task during training to have its own independent dataset. During the learning cycle of a task, only samples from that task are input, and samples from different tasks are independent and identically distributed. In other words, these methods assume that one task is trained at a time, with a hard boundary between tasks. Each task is trained using a complete dataset related to that task, and training continues with the next task after completion. Training for each task is performed offline, not online. However, in real-world applications, data arrives in a streaming manner, making task-based usage rare.

[0060] It is precisely based on the above problems that this application provides a classification method based on multi-perspective continuous learning. The proposed CAEE (Coupled AutoEncoders and Ensemble) model is task-free and has a wider range of application scenarios. In this method, the acquired first-perspective image data and second-perspective image data are fused to obtain dual-perspective fusion data, which is then input into multiple pre-learned classifiers. Based on the probability of the category to which the dual-perspective fusion data output by each classifier belongs, the final classification result is aggregated. This solves the problem that the fusion process of the traditional deep learning model has a certain degree of randomness, the model cannot remember or recognize the previous state, and it is difficult to handle the problem of revisiting the previous state, thereby further improving the safety and stability of the unmanned driving system in complex scenarios.

[0061] Specifically, Figure 1 A flowchart of a classification method based on multi-perspective continuous learning provided in an embodiment of the present application.

[0062] Before introducing the classification method based on multi-perspective continuous learning proposed in the embodiment of the present application, the technical solution adopted in the embodiment of the present application is first introduced.

[0063] This application proposes a CAEE model, which consists of a pair of pre-trained autoencoders (Coupled AutoEncoder, referred to as CAE) and a set of simple single-layer classifiers, such as Figure 2 As shown in Figure 3, the trained CAE is used to fuse data from two perspectives and pass the fused implicit representation to the corresponding classifier. The selected classifier outputs the probability of the category to which the implicit representation belongs, and finally aggregates the output results of all classifiers to obtain the final classification result.

[0064] Among them, such as Figure 3 As shown in (a) to (d), Figure 3 (a) is the Sihouette distribution diagram of samples in each cluster, Figure 3(b) is a schematic diagram of the latent space generated by AE. Figure 3 (c) is the Sihouette distribution diagram of samples in each cluster, Figure 3 (d) is a schematic diagram of the latent space generated by CAE. The trained CAE is integrated with multiple perspectives to obtain a latent space with clear clustering (i.e., implicit representation). During the training stage, the embodiment of the present application corresponds to multiple simple classifiers in the latent space generated by CAE, and ultimately different classifiers can be selected for different classification tasks, so that some classifiers can be trained more frequently on data of a certain category and focus on that category, and will no longer be selected to learn new tasks, thereby achieving the purpose of combating forgetting.

[0065] It should be noted that the embodiment of the present application is divided into three modules based on the multi-perspective fusion module: self-reconstruction, cross-perspective reconstruction, and consistency constraint. Among them, the self-reconstruction module captures the complete information of each perspective; the cross-perspective reconstruction module migrates the information obtained from one perspective to another; and the consistency constraint module makes the encoding of different perspectives more consistent. In addition, consistency constraints on perspective encoding are also conducive to eliminating the randomness of different encoders and improving the decoder's ability to express different perspectives.

[0066] Specifically, if Figure 1 As shown, the classification method based on multi-view continuous learning includes the following steps:

[0067] In step S101 , first-perspective image data and second-perspective image data are acquired.

[0068] Among them, the first perspective image data is Noisy-MNIST perspective image data, and the second perspective image data is Half-MNIST perspective image data. The embodiment of the present application can collect image data through a vehicle-mounted sensor.

[0069] In step S102, the first-perspective image data and the second-perspective image data are fused to obtain dual-perspective fusion data, and the dual-perspective fusion data is input into multiple pre-learned classifiers to obtain the probability of the category to which the dual-perspective fusion data output by each classifier belongs.

[0070] Among them, the embodiment of the present application uses CAE to fuse data from two perspectives, and passes the fused implicit representation, i.e., dual-perspective fusion data, to the corresponding classifier, and the selected classifier will output the probability of the category to which the implicit representation belongs.

[0071] Furthermore, in some embodiments, before inputting the dual-view fusion data into a plurality of pre-learned classifiers, it also includes: obtaining a data set to be trained, wherein the data set to be trained includes first-view data to be trained and second-view data to be trained; using a pre-trained autoencoder to fuse the first-view data to be trained and the second-view data to be trained to obtain a fused implicit representation; based on a preset minimization loss function, using the fused implicit representation to train an initial classification model to obtain a plurality of pre-learned classifiers, wherein the initial classification model is composed of a plurality of single-layer classifiers and multi-view image data.

[0072] Specifically, the dataset used in the embodiment of the present application is a more challenging dual-view dataset generated from the classic MNIST dataset, where the first-view image data is the original MNIST image. In order to generate the second-view image data, the embodiment of the present application uniformly adds independent random noise to each pixel of each first-view image, and finally scales the pixel value to the interval [0,1].

[0073] Furthermore, in some embodiments, obtaining a dataset to be trained includes: obtaining a Noisy-MNIST perspective image dataset and a Half-MNIST perspective image dataset; and dividing both the Noisy-MNIST perspective image dataset and the Half-MNIST perspective image dataset into a dataset to be trained consisting of a preset number of images according to a preset segmentation strategy.

[0074] The preset number of images constituting the dataset to be trained may be a number pre-set by the user, a number obtained through a finite number of experiments, or a number obtained through a finite number of computer simulations, and is not specifically limited here.

[0075] Furthermore, if Figure 4 As shown, the embodiment of the present application divides the dual-view dataset into a training dataset consisting of 10,000 images, a verification dataset consisting of 2,000 images, and a test dataset consisting of 6,000 images.

[0076] Furthermore, in some embodiments, the preset minimization loss function is:

[0077]

[0078] in, are the training samples of the first and second perspectives respectively, y t is the label corresponding to the sample, Θ=[θ1,θ2,...θ n ] represents the parameters of the N single-layer classifiers that constitute the continuous learning module.

[0079] Specifically, the goal of the CAEE model proposed in the embodiment of the present application is to learn a classification model f(x 1 ,x 2 ; Θ), so that the model can complete all the classification tasks implicit in D, that is, the model needs to continuously learn the image samples with new classification labels emerging in D, where x 1 ,x 2 They are training samples from the first and second perspectives, and each sample has a label y corresponding to it, Θ=[θ1,θ2,...θ n ] represents the parameters of the N single-layer classifiers that constitute the continuous learning module. At each time step t in the training process, the model will pass the newly accepted sample (x t ,y t ) to minimize the loss function And the loss before t is not increased.

[0080] In step S103, the final classification result is obtained by aggregating the probabilities of the categories to which the dual-view fusion data output by each classifier belongs.

[0081] In order to facilitate those skilled in the art to further understand the classification method based on multi-view continuous learning proposed in this embodiment, the following is combined with Figure 5 Elaborate in detail.

[0082] Specifically, the embodiment of the present application uses a computer with an Intel(R) Core(TM) i9-7900X processor and two NVIDIA2080Ti graphics cards to evaluate the CAEE model on the Noisy-MNIST and Half-MNIST datasets. The test results are shown in Tables 1 and 2.

[0083] Table 1

[0084]

[0085] Table 2

[0086]

[0087] It is understandable that the embodiment of this application divides the Noisy-MNIST dataset into 5 equal parts, and the performance of the CAEE model is undoubtedly the best. Figure 5As shown in (a), if a common classifier (such as the Vanilla model) is used to learn these five tasks, the classifier needs to be retrained each time a task is learned, which will lead to catastrophic forgetting. The curve below shows the performance of the Vanilla model. After the first task training is completed, the performance of the Vanilla model on this task reaches more than 85%. However, when the training task switches to the second task, the performance of the Vanilla model on the first task drops rapidly; Figure 5 As shown in (b), its performance on the second task increases rapidly.

[0088] This shows that ordinary classifiers, when trained using traditional machine learning methods, tend to forget past knowledge when fitting new datasets, resulting in catastrophic forgetting. Moreover, as the training time increases and the number of subsequent tasks increases, the forgetting of previous tasks will be further aggravated.

[0089] The CAEE model proposed in the embodiment of the present application can combat the aforementioned catastrophic forgetting, such as Figure 5 As shown in (a), the upper curve shows the performance of the CAEE model. After the first task training is completed, the CAEE model also achieves an excellent performance of nearly 100% accuracy on this task. As the task switches, Figure 5 As shown in (b), the performance of the CAEE model on this task only slowly decreases until the last task training is completed. Its performance on the initial task 1 still remains above 90%, which well preserves the previous knowledge. Moreover, the model performs stably in the process of multiple runs with a very small error range, as shown in Figure 5 As shown in (c), the overall performance comparison between the CAEE model and the Vanilla model on the Noisy-MNIST dataset fully demonstrates the effectiveness of the CAEE model.

[0090] Regarding the ability to combat forgetting, by comparing the CAEE model with the regularization-based method EWC (Either-Way Communication, half-duplex communication) and the memory replay-based method IcaRL, and observing the forgetting levels of the three in Tables 1 and 2, it can be found that the CAEE model performs better than the EWC and IcaRL models in combating forgetting on the two selected datasets. As the task switches, the CAEE model has the smallest change in accuracy on the previous task, which is particularly evident on the Half-Mnist dataset.

[0091] It is understandable that since both the EWC and IcaRL models are single-perspective, the embodiment of the present application only uses the data from one of the perspectives in the dataset as input, but the amount of information contained in the two perspectives is average. Therefore, the EWC and IcaRL models can only be inferred through half of the data in the original dataset. At this time, their accuracy rates are only 66.91% and 63.33%, respectively, which is significantly lower than the high accuracy of 81.22% of the CAEE model with a multi-perspective fusion module.

[0092] Furthermore, in the more challenging task-free infinite data flow scenario (corresponding to the Gaussian planning in Tables 1 and 2), although the limitations of the method selected in the embodiment of the present application result in no direct control group in this scenario, by comparing the experimental results with the same data set with 5-way segmentation, it can be found that the performance of the CAEE model in the experimental scenario of task-free infinite data flow is similar to that in the 5-way segmentation scenario, indicating that the CAEE model proposed in the embodiment of the present application is effective for task-free continuous learning.

[0093] It should be noted that the CAEE model proposed in the embodiment of the present application is composed of an encoder with many parameters and many simple classifiers. All these classifiers will also occupy a large amount of memory. The embodiment of the present application will try to make more efficient use of memory space on the basis of better combating forgetting.

[0094] The multi-perspective continuous learning classification method proposed in this application fuses acquired first-perspective image data with second-perspective image data to generate dual-perspective fused data. This dual-perspective fused data is then fed into multiple pre-learned classifiers. Based on the probabilities of the categories to which the dual-perspective fused data output by each classifier belongs, the final classification result is aggregated. This overcomes the inherent randomness of the fusion process in traditional deep learning models, which hinders the model's ability to remember or recognize previous states, making it difficult to revisit previous states. This further improves the safety and stability of autonomous driving systems in complex scenarios.

[0095] Next, a classification device based on multi-perspective continuous learning proposed in an embodiment of the present application is described with reference to the accompanying drawings.

[0096] Figure 6 It is a block diagram of a classification device based on multi-view continuous learning according to an embodiment of the present application.

[0097] like Figure 6 As shown, the classification device 10 based on multi-view continuous learning includes: an acquisition module 100, a processing module 200 and an aggregation module 300.

[0098] The acquisition module 100 is used to acquire the first-view image data and the second-view image data;

[0099] a processing module 200 for fusing the first-view image data and the second-view image data to obtain dual-view fused data, and inputting the dual-view fused data into a plurality of pre-learned classifiers to obtain the probability of the category to which the dual-view fused data output by each classifier belongs; and

[0100] The aggregation module 300 is used to aggregate the probability of the category to which the dual-view fusion data output by each classifier belongs to obtain a final classification result.

[0101] Furthermore, in some embodiments, before inputting the dual-view fusion data into a plurality of pre-learned classifiers, the processing module 200 is further configured to:

[0102] Acquire a dataset to be trained, wherein the dataset to be trained includes first-perspective training data and second-perspective training data;

[0103] Use the pre-trained autoencoder to fuse the first-perspective training data and the second-perspective training data to obtain the fused implicit representation;

[0104] Based on a preset minimization loss function, the fused implicit representation is used to train the initial classification model to obtain multiple pre-learned classifiers, where the initial classification model consists of multiple single-layer classifiers and multi-view image data.

[0105] Furthermore, in some embodiments, the processing module 200 obtains a dataset to be trained, and is specifically configured to:

[0106] Obtain the Noisy-MNIST and Half-MNIST perspective image datasets;

[0107] According to the preset segmentation strategy, both the Noisy-MNIST perspective image dataset and the Half-MNIST perspective image dataset are divided into training datasets consisting of a preset number of images.

[0108] Furthermore, in some embodiments, the preset minimization loss function is:

[0109]

[0110] in, are the training samples of the first and second perspectives respectively, y t is the label corresponding to the sample, Θ=[θ1,θ2,...θ n ] represents the parameters of the N single-layer classifiers that constitute the continuous learning module.

[0111] It should be noted that the aforementioned explanation of the embodiment of the classification method based on multi-view continuous learning is also applicable to the classification device based on multi-view continuous learning in this embodiment, and will not be repeated here.

[0112] The multi-perspective continuous learning classification device proposed in the embodiments of this application fuses acquired first-perspective image data with second-perspective image data to generate dual-perspective fused data. This dual-perspective fused data is then fed into multiple pre-learned classifiers. Based on the probabilities of the categories to which the dual-perspective fused data output by each classifier belongs, the final classification result is aggregated. This solves the problem of the fusion process in traditional deep learning models being somewhat random, the model being unable to remember or recognize previous states, and the resulting difficulty in revisiting previous states. This further improves the safety and stability of autonomous driving systems in complex scenarios.

[0113] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0114] Memory 701 , processor 702 , and computer programs stored in the memory 701 and executable on the processor 702 .

[0115] When the processor 702 executes the program, the classification method based on multi-view continuous learning provided in the above embodiment is implemented.

[0116] Furthermore, the electronic device further includes:

[0117] The communication interface 703 is used for communication between the memory 701 and the processor 702 .

[0118] The memory 701 is used to store computer programs that can be run on the processor 702 .

[0119] The memory 701 may include a high-speed RAM (Random Access Memory) memory, and may also include a non-volatile memory, such as at least one disk memory.

[0120] If the memory 701, processor 702, and communication interface 703 are implemented independently, the communication interface 703, memory 701, and processor 702 can be connected to each other via a bus and communicate with each other. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0121] Optionally, in a specific implementation, if the memory 701, the processor 702 and the communication interface 703 are integrated on a chip, the memory 701, the processor 702 and the communication interface 703 can communicate with each other through an internal interface.

[0122] The processor 702 may be a CPU (Central Processing Unit), or an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement the embodiments of the present application.

[0123] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned classification method based on multi-perspective continuous learning.

[0124] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0125] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of this application, "N" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0126] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or more executable instructions for implementing the steps of a custom logical function or process, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application belong.

[0127] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiment, the N steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array, a field programmable gate array, etc.

[0128] Those skilled in the art will appreciate that all or part of the steps in the method for implementing the above-mentioned embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0129] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A classification method based on multi-view continuous learning, characterized in that: The following steps are involved: Acquire first-perspective image data and second-perspective image data; fusing the first-view image data and the second-view image data to obtain dual-view fusion data, and inputting the dual-view fusion data into a plurality of pre-learned classifiers to obtain a probability of the category to which the dual-view fusion data belongs, output by each classifier; as well as Based on the probability of the category to which the dual-view fusion data output by each classifier belongs, aggregating to obtain a final classification result; Before inputting the dual-view fusion data into the pre-learned multiple classifiers, the method further includes: Acquire a data set to be trained, wherein the data set to be trained includes first-perspective training data and second-perspective training data; Using a pre-trained autoencoder to fuse the first-perspective training data and the second-perspective training data to obtain a fused implicit representation; Based on a preset minimization loss function, the fused implicit representation is used to train an initial classification model to obtain the pre-learned multiple classifiers, wherein the initial classification model is composed of multiple single-layer classifiers and multi-view image data; the preset minimization loss function is: in, are the training samples of the first and second perspectives respectively, y t is the label corresponding to the sample, Θ=[θ1,θ2,...θ n ] represents the parameters of the N single-layer classifiers that constitute the continuous learning module.

2. The method according to claim 1, characterized in that The step of obtaining a dataset to be trained includes: Obtain the Noisy-MNIST and Half-MNIST perspective image datasets; According to a preset segmentation strategy, the Noisy-MNIST perspective image dataset and the Half-MNIST perspective image dataset are both segmented into the to-be-trained dataset consisting of a preset number of images.

3. A classification device based on multi-view continuous learning, characterized in that: include: An acquisition module, configured to acquire first-perspective image data and second-perspective image data; a processing module, configured to fuse the first-view image data and the second-view image data to obtain dual-view fusion data, and input the dual-view fusion data into a plurality of pre-learned classifiers to obtain a probability of the category to which the dual-view fusion data belongs, as output by each classifier; as well as an aggregation module, configured to aggregate the probabilities of the categories to which the dual-view fusion data output by each classifier belongs to obtain a final classification result; and before inputting the dual-view fusion data into the pre-learned multiple classifiers, the processing module is further configured to: Acquire a data set to be trained, wherein the data set to be trained includes first-perspective training data and second-perspective training data; Using a pre-trained autoencoder to fuse the first-perspective training data and the second-perspective training data to obtain a fused implicit representation; Based on a preset minimization loss function, the fused implicit representation is used to train an initial classification model to obtain the pre-learned multiple classifiers, wherein the initial classification model is composed of multiple single-layer classifiers and multi-view image data; the preset minimization loss function is: in, are the training samples of the first and second perspectives respectively, y t is the label corresponding to the sample, Θ=[θ1,θ2,...θ n ] represents the parameters of the N single-layer classifiers that constitute the continuous learning module.

4. The device according to claim 3, characterized in that The processing module is used to obtain a training data set and to: Obtain the Noisy-MNIST and Half-MNIST perspective image datasets; According to a preset segmentation strategy, the Noisy-MNIST perspective image dataset and the Half-MNIST perspective image dataset are both segmented into the to-be-trained dataset consisting of a preset number of images.

5. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the classification method based on multi-view continuous learning according to any one of claims 1 to 2.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the classification method based on multi-view continuous learning as described in any one of claims 1-2.

Images