Multi-modal automatic labeling method, labeling model training method, and related device

Abstract

The embodiment of the application discloses a multi-modal automatic labeling method, a labeling model training method and related equipment, realizes the interaction of multi-modal data in space and time, utilizes richer information to realize task automatic labeling, and is beneficial to improving the accuracy of the final labeling result. The embodiment of the application comprises the following steps: acquiring first data and second data of different modes corresponding to a first time; performing attention mechanism-based processing on the first data, the second data, a first historical feature and a second historical feature to obtain a first space-time feature and a second space-time feature corresponding to the first time, the first space-time feature corresponding to the first data, and the second space-time feature corresponding to the second data; performing feature fusion on the first space-time feature and the second space-time feature corresponding to the first time to obtain a multi-modal feature corresponding to the first time; and performing labeling processing based on the multi-modal feature to output a labeling result corresponding to the first time.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a multi-modal automatic labeling method, a labeling model training method, and related equipment. BACKGROUND

[0002] Artificial intelligence (AI) is the use of digital computers or digital computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that aims to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is the design principle and implementation method of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making. Using artificial intelligence to automatically label data is a common application in the field of artificial intelligence.

[0003] The existing data automatic labeling method mainly relies on single modal input to complete a single labeling task (such as three-dimensional target frame labeling based on point cloud), and cannot fully utilize the consistent information between multi-modal data for collaborative optimization, resulting in insufficient precision of the final labeling result. SUMMARY

[0004] The embodiments of the present application provide a multi-modal automatic labeling method, a labeling model training method, and related equipment. By fusing unlabeled multi-modal data based on cross-modal and cross-temporal attention mechanisms, unified multi-modal features are obtained, the interaction of multi-modal data in space and time is realized, richer information is used to realize task automatic labeling, and the accuracy of the final labeling result is improved.

[0005] In a first aspect, the present application provides a multi-modal automatic labeling method, which comprises:

[0006] obtaining first data and second data corresponding to a first time, the first data and the second data being data of different modalities;

[0007] The first data, the second data, the first historical feature, and the second historical feature are processed based on an attention mechanism to obtain a first spatiotemporal feature and a second spatiotemporal feature corresponding to the first moment, wherein the first historical feature and the second historical feature are respectively the first spatiotemporal feature and the second spatiotemporal feature corresponding to a second moment, the second moment is a moment before the first moment, the first spatiotemporal feature corresponds to the first data, and the second spatiotemporal feature corresponds to the second data; the processing process combines historical features, that is, features corresponding to moments before the current moment are interactively processed, and therefore, the features obtained in this application realize cross-time feature interaction. In addition, the data input by the embodiments of this application is multi-modal data at the current moment, and the multi-modal data is interactively processed in the output feature process, and therefore, the features obtained in this application also realize cross-space feature interaction.

[0008] The first spatiotemporal feature and the second spatiotemporal feature corresponding to the first moment are fused to obtain a multi-modal feature corresponding to the first moment.

[0009] The multi-modal feature is processed based on labeling to output a labeling result corresponding to the first moment.

[0010] In this application, feature fusion is performed on multi-modal input data to realize data labeling. After obtaining the first data and the second data of different modalities, the first data, the second data, the first historical feature, and the second historical feature are processed based on an attention mechanism to obtain corresponding spatiotemporal features, and the spatiotemporal features are fused to obtain a multi-modal feature. The unified multi-modal feature after fusion can be used as input to a labeling task for labeling processing, thereby obtaining a labeling result corresponding to the first moment. By fusing multi-modal data based on cross-modal and cross-spatiotemporal attention mechanisms, the interaction of multi-modal data in space and time is realized, richer information is used to realize task automatic labeling, and the accuracy of the final labeling result is improved.

[0011] In a possible implementation method, the first data, the second data, the first historical feature, and the second historical feature are processed based on an attention mechanism to obtain a first spatiotemporal feature and a second spatiotemporal feature corresponding to the first moment, and the processing specifically includes:

[0012] The first data and the second data are respectively processed based on a self-attention mechanism to obtain a first spatial feature and a second spatial feature;

[0013] The first spatial feature and the second spatial feature are interactively processed based on an attention mechanism to obtain a first interaction feature and a second interaction feature, wherein the first interaction feature corresponds to the first data, and the second interaction feature corresponds to the second data;

[0014] The first interaction feature and the first historical feature are processed based on a cross-attention mechanism to obtain a first space-time feature corresponding to the first moment;

[0015] The second interaction feature and the second historical feature are processed based on a cross-attention mechanism to obtain a second space-time feature corresponding to the first moment.

[0016] In the present application, first, the first data and the second data are processed based on a self-attention mechanism to capture the relevance between different positions or elements in the data; the two space features are interactively processed based on a cross-attention mechanism, so that the two features have a spatial concept; after the spatial interaction, the historical feature is interacted to obtain a space-time feature.

[0017] In a possible implementation method, the method further includes:

[0018] Based on a multi-layer perceptron, at least one of the first space feature, the second space feature, the first interaction feature, the second interaction feature, the first space-time feature, or the second space-time feature is enhanced.

[0019] In the present application, after the input data is processed based on the self-attention mechanism or the cross-attention mechanism, it can be input into the multi-layer perceptron for enhancement.

[0020] In a possible implementation method, the first interaction feature and the first historical feature are processed based on a cross-attention mechanism to obtain a first space-time feature corresponding to the first moment; the second interaction feature and the second historical feature are processed based on a cross-attention mechanism to obtain a second space-time feature corresponding to the first moment, specifically including:

[0021] The first space feature, the first interaction feature, and the first historical feature are processed based on a cross-attention mechanism to obtain a first space-time feature corresponding to the first moment;

[0022] The second space feature, the second interaction feature, and the second historical feature are processed based on a cross-attention mechanism to obtain a second space-time feature corresponding to the first moment.

[0023] In the present application, through the way of jump connection, the space feature processed by the self-attention layer can be directly transmitted to the subsequent time cross-attention layer, which helps to stabilize the training process and improve the convergence and generalization ability of the model.

[0024] In a possible implementation method, the multi-modal feature is processed based on a multi-modal feature to output a labeling result, specifically including:

[0025] The multi-modal feature is marked by the first marking model and the second marking model respectively to obtain corresponding first marking information and second marking information, and the first marking model and the second marking model are different marking models.

[0026] The first marking information and the second marking information are interacted to output a marking result.

[0027] In the application, the unified multi-modal feature representation is input into a multi-task marking module for marking processing, and the model output information of each module is interacted to obtain a final result, thereby improving the accuracy of the output marking result of the automatic marking system.

[0028] In a possible implementation method, the first marking information and the second marking information are interacted to output a marking result, specifically including:

[0029] The second marking information is input into the first marking model as prior information of the multi-modal feature to obtain a first marking result.

[0030] The first marking information is input into the second marking model as prior information of the multi-modal feature to obtain a second marking result.

[0031] The marking result includes the first marking result and the second marking result.

[0032] In the application, the first marking information and the second marking information are interacted, specifically, the output results of different marking task modules are interacted as prior information of other marking tasks.

[0033] In a possible implementation method, the first marking information and the second marking information are interacted to output a marking result, specifically including:

[0034] The prompt information is generated by a prompt information encoder according to the first marking information and the second marking information.

[0035] The prompt information is input into the first marking model and the second marking model respectively to obtain corresponding first marking results and second marking results.

[0036] The marking result includes the first marking result and the second marking result.

[0037] In the embodiment of the application, the first marking information and the second marking information can also be input into the prompt information encoder as cross-task automatic prompt information, and then the prompt information is input into the multi-task marking module to refine the marking result of each module to obtain a corresponding marking result.

[0038] In a possible implementation method, the labeling model includes a three-dimensional (3D) target box detection model, a semantic segmentation model, a target tracking model, and a behavior analysis model.

[0039] In a second aspect, the application provides a labeling model training method, including:

[0040] obtaining first data and second data corresponding to the first moment, the first data and the second data being data of different modalities;

[0041] processing the first data and the second data by using a to-be-trained model to obtain a labeling result corresponding to the first moment, wherein the to-be-trained model is configured to: perform attention mechanism-based processing on the first data, the second data, a first historical feature, and a second historical feature to obtain a first spatio-temporal feature and a second spatio-temporal feature corresponding to the first moment, the first historical feature and the second historical feature being the first spatio-temporal feature and the second spatio-temporal feature corresponding to a second moment respectively, the second moment being a moment before the first moment, the first spatio-temporal feature corresponding to the first data, and the second spatio-temporal feature corresponding to the second data; perform feature fusion on the first spatio-temporal feature and the second spatio-temporal feature corresponding to the first moment to obtain a multi-modal feature; and perform labeling processing based on the multi-modal feature to output the labeling result corresponding to the first moment;

[0042] training the to-be-trained model based on the labeling result to obtain a target model.

[0043] In a possible implementation method, the attention mechanism-based processing on the first data, the second data, the first historical feature, and the second historical feature to obtain the first spatio-temporal feature and the second spatio-temporal feature corresponding to the first moment specifically includes:

[0044] performing self-attention mechanism-based processing on the first data and the second data respectively to obtain a first spatial feature and a second spatial feature;

[0045] performing attention mechanism-based interaction processing on the first spatial feature and the second spatial feature to obtain a first interaction feature and a second interaction feature, wherein the first interaction feature corresponds to the first data, and the second interaction feature corresponds to the second data;

[0046] performing cross-attention mechanism-based processing on the first interaction feature and the first historical feature to obtain the first spatio-temporal feature corresponding to the first moment;

[0047] performing cross-attention mechanism-based processing on the second interaction feature and the second historical feature to obtain the second spatio-temporal feature corresponding to the first moment.

[0048] In a possible implementation method, the method further includes:

[0049] The first spatial feature, the second spatial feature, the first interaction feature, the second interaction feature, the first spatio-temporal feature, or the second spatio-temporal feature are enhanced based on a multi-layer perceptron.

[0050] In a possible implementation method, the first interaction feature and the first historical feature are processed based on a cross-attention mechanism to obtain the first spatio-temporal feature corresponding to the first moment; and the second interaction feature and the second historical feature are processed based on a cross-attention mechanism to obtain the second spatio-temporal feature corresponding to the first moment, specifically including:

[0051] The first spatial feature, the first interaction feature, and the first historical feature are processed based on a cross-attention mechanism to obtain the first spatio-temporal feature corresponding to the first moment;

[0052] The second spatial feature, the second interaction feature, and the second historical feature are processed based on a cross-attention mechanism to obtain the second spatio-temporal feature corresponding to the first moment.

[0053] In a third aspect, the present application provides a multi-modal automatic labeling device, comprising:

[0054] An input layer is configured to obtain first data and second data corresponding to a first moment, the first data and the second data being data of different modalities;

[0055] A multi-modal spatio-temporal attention layer is configured to process the first data, the second data, a first historical feature, and a second historical feature based on an attention mechanism to obtain a first spatio-temporal feature and a second spatio-temporal feature corresponding to the first moment, wherein the first historical feature and the second historical feature are respectively a first spatio-temporal feature and a second spatio-temporal feature corresponding to a second moment, the second moment being a moment before the first moment, the first spatio-temporal feature corresponding to the first data, and the second spatio-temporal feature corresponding to the second data;

[0056] A spatio-temporal feature fusion layer is configured to perform feature fusion on the first spatio-temporal feature and the second spatio-temporal feature corresponding to the first moment to obtain a multi-modal feature corresponding to the first moment;

[0057] A task labeling layer is configured to perform labeling processing based on the multi-modal feature to output a labeling result corresponding to the first moment.

[0058] In a possible implementation method,

[0059] The multi-modal spatio-temporal attention layer is specifically configured to perform self-attention mechanism-based processing on the first data and the second data respectively to obtain first spatial features and second spatial features; perform attention mechanism-based interaction processing on the first spatial features and the second spatial features to obtain first interaction features and second interaction features, wherein the first interaction features correspond to the first data, and the second interaction features correspond to the second data; perform cross-attention mechanism-based processing on the first interaction features and the first historical features to obtain first spatio-temporal features corresponding to the first time; and perform cross-attention mechanism-based processing on the second interaction features and the second historical features to obtain second spatio-temporal features corresponding to the first time.

[0060] In a possible implementation method,

[0061] The task labeling layer is specifically configured to perform labeling processing on the multi-modal features by using a first labeling model and a second labeling model respectively to obtain corresponding first labeling information and second labeling information, the first labeling model and the second labeling model being different labeling models; and perform information interaction on the first labeling information and the second labeling information to output a labeling result.

[0062] In a fourth aspect, the present application provides a labeling model training device, comprising:

[0063] The acquisition module is configured to acquire first data and second data corresponding to a first time, the first data and the second data being data of different modalities;

[0064] The generation module is configured to perform processing on the first data and the second data by using a to-be-trained model to obtain a labeling result corresponding to the first time, wherein the to-be-trained model is configured to: perform attention mechanism-based processing on the first data, the second data, first historical features, and second historical features to obtain first spatio-temporal features and second spatio-temporal features corresponding to the first time, the first historical features and the second historical features being the first spatio-temporal features and the second spatio-temporal features corresponding to a second time, the second time being a time before the first time, the first spatio-temporal features corresponding to the first data, and the second spatio-temporal features corresponding to the second data; perform feature fusion on the first spatio-temporal features and the second spatio-temporal features corresponding to the first time to obtain multi-modal features; and perform labeling processing based on the multi-modal features to output the labeling result corresponding to the first time;

[0065] The training module is configured to train the to-be-trained model based on the labeling result to obtain a target model.

[0066] In a fifth aspect, the present application provides a computer program, which, when running on a computer, causes the computer to execute the method of any one of the first aspect, or causes the computer to execute the method of any one of the second aspect.

[0067] In a sixth aspect, the present application provides a computer readable storage medium, comprising a program, when the program is executed on a computer, causing the computer to execute the method of any one of the first aspect, or causing the computer to execute the method of any one of the second aspect.

[0068] In a seventh aspect, the present application provides an execution device, comprising a processor and a memory, the processor being coupled with the memory,

[0069] the memory being configured to store a program;

[0070] the processor being configured to execute the program in the memory, so that the execution device executes the method of any one of the first aspect.

[0071] In an eighth aspect, the present application provides a training device, comprising a processor and a memory, the processor being coupled with the memory,

[0072] the memory being configured to store a program;

[0073] the processor being configured to execute the program in the memory, so that the training device executes the method of any one of the second aspect.

[0074] The schemes provided by the second aspect to the eighth aspect are used to realize or cooperate to realize the method provided by the first aspect, and thus can achieve the same or corresponding beneficial effects as the first aspect. Here, no further description is given. BRIEF DESCRIPTION OF DRAWINGS

[0075] Figure 1 A structural schematic diagram of an artificial intelligence subject framework;

[0076] Figure 2a A structural schematic diagram of an automatic labeling system provided by an embodiment of the present application;

[0077] Figure 2b Another structural schematic diagram of an automatic labeling system provided by an embodiment of the present application;

[0078] Figure 3 A schematic diagram of a system 100 architecture provided by an embodiment of the present application;

[0079] Figure 4 A flow schematic diagram of a multi-modal automatic labeling method provided by an embodiment of the present application;

[0080] Figure 5 A possible implementation method of a corresponding embodiment; Figure 4 A flowchart of a possible implementation method of a corresponding embodiment;

[0081] Figure 6This is a schematic diagram of the structure of a cross-modal spatiotemporal Transformer block provided in an embodiment of this application;

[0082] Figure 7 This is a schematic diagram of another cross-modal spatiotemporal Transformer block provided in the embodiments of this application;

[0083] Figure 8 This is a schematic diagram of the cross-modal spatiotemporal feature fusion layer provided in an embodiment of this application;

[0084] Figure 9 This is an architecture diagram of the multimodal automatic annotation system provided in the embodiments of this application;

[0085] Figure 10 This is a schematic diagram of the structure of the multi-task annotation module provided in the embodiments of this application;

[0086] Figure 11 For based on Figure 10 The diagram shows the architecture of a multimodal automatic annotation system with a multi-task annotation module.

[0087] Figure 12 A flowchart illustrating a training method for an labeled model provided in an embodiment of this application;

[0088] Figure 13 A schematic diagram of the structure of the multimodal automatic annotation device provided in the embodiments of this application;

[0089] Figure 14 A schematic diagram of a training device for an labeled model provided in an embodiment of this application;

[0090] Figure 15 A schematic diagram of the structure of the execution device provided in the embodiments of this application;

[0091] Figure 16 This is a schematic diagram of a training device provided in an embodiment of this application. Detailed Implementation

[0092] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application are described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Those skilled in the art will understand that with the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0093] The terms "first", "second", and the like in the description and in the claims of the present application and above-described drawings are used to distinguish similar objects, and are not necessarily used to describe a particular sequential or chronological order. It should be understood that the data thus used can be interchanged, where appropriate, so that the embodiments described herein can be carried out in sequences other than those illustrated or described herein. Moreover, the terms "comprise" and "have", and any variations thereof, are intended to cover a non-exclusive inclusion, for example, a process, method, system, product, or apparatus that comprises a list of steps or modules as non-limiting to those steps or modules that are expressly listed, but can include other steps or modules not expressly listed or inherent to such process, method, product, or apparatus. The naming or numbering of steps in the present application does not mean that the steps in the method flow must be executed in the time / logical order indicated by the naming or numbering, and the named or numbered flow steps can change the execution order according to the technical purpose to be achieved, as long as the same or similar technical effects can be achieved. The division of units in the present application is a logical division, and in actual application, it can have another division method, for example, multiple units can be combined or integrated into another system, or some features can be ignored or not executed, in addition, the coupling or direct coupling or communication connection between the units shown or discussed can be through some interface, the indirect coupling or communication connection between the units can be electrical or other similar forms, which are not limited in the present application. And the units or sub-units described as separate components can or can not be physically separate, can or can not be physical units, or can be distributed to multiple circuit units, and some or all of the units can be selected according to actual needs to achieve the purpose of the present application.

[0094] Automatic labeling technology refers to using computer algorithms and machine learning methods to input raw data (such as pictures, text, voice, automatic driving, etc.) into a computer, and using a computer to automatically classify, label, annotate, etc. Process raw data to facilitate subsequent data analysis and application. Common data labeling tasks include image classification, target detection, speech recognition, natural language processing, etc.

[0095] Data labeling technology is widely used in the field of autonomous driving. In the past, data labeling mainly relied on manual labeling. With the continuous collection of massive unlabeled autonomous driving data, manual labeling alone cannot meet the demand, which restricts the iterative update of autonomous driving models and commercial landing. Therefore, using artificial intelligence for automatic data labeling is a common application in the field of artificial intelligence. Existing automatic data labeling methods for autonomous driving mainly rely on single modal input to complete single labeling tasks (such as three-dimensional target frame labeling based on point cloud), which cannot fully utilize the consistent information between multi-modal autonomous driving data for collaborative optimization, nor can it fully utilize the complementary information between multiple labeling tasks for multi-functional optimization. Therefore, designing a multi-functional automatic labeling system based on multi-modal autonomous driving data has high value.

[0096] First, the overall workflow of the artificial intelligence system is described, please refer to Figure 1 , Figure 1 is a structural diagram of the artificial intelligence subject framework. The following describes the above artificial intelligence subject framework from two dimensions of "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis). Among them, "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, data undergoes a condensation process of "data-information-knowledge-wisdom". "IT value chain" reflects the value brought by artificial intelligence to the information technology industry from the bottom infrastructure of artificial intelligence, information (provision and processing technology implementation) to the industrial ecological process of the system.

[0097] (1) Infrastructure

[0098] Infrastructure provides computing power support for artificial intelligence systems, enables communication with the outside world, and realizes support through the underlying platform. Communication with the outside world through sensors; computing power is provided by intelligent chips (CPU, NPU, GPU, ASIC, FPGA, etc. Hardware acceleration chips); the underlying platform includes distributed computing framework and network-related platform support and support, which can include cloud storage and computing, interconnection network, etc. For example, sensors and external communication acquire data, which are provided to intelligent chips in the distributed computing system of the underlying platform for calculation.

[0099] (2) Data

[0100] The data on the upper layer of the infrastructure is used to represent the data source in the field of artificial intelligence. Data involves graphics, images, speech, text, and also involves Internet of Things data of traditional devices, including business data of existing systems and sensing data such as force, displacement, liquid level, temperature, and humidity.

[0101] (3) Data processing

[0102] Data processing generally includes data training, machine learning, deep learning, search, reasoning, decision, etc.

[0103] Among them, machine learning and deep learning can model, extract, preprocess, train, etc. symbolic and formalized intelligent information on data.

[0104] Reasoning refers to the process of simulating human intelligent reasoning methods in computers or intelligent systems, using formalized information to solve problems according to reasoning control strategies, and the typical function is search and matching.

[0105] Decision refers to the process of decision-making after intelligent information is reasoned, and usually provides functions such as classification, sorting, and prediction.

[0106] (4) General capabilities

[0107] After data is processed as mentioned above, some general capabilities can be formed based on the results of data processing, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0108] (5) Intelligent products and industry applications

[0109] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields, which are the packaging of overall solutions of artificial intelligence, and realize the application of intelligent information decision-making products. The application fields mainly include intelligent terminals, intelligent transportation, intelligent medical treatment, automatic driving, smart city, etc.

[0110] Next, several application scenarios of the present application are introduced.

[0111] Figure 2a A structural schematic diagram of an automatic labeling system provided by an embodiment of the present application is shown, which includes a user device and a data processing device, wherein the user device includes a mobile phone, a personal computer, or an intelligent terminal such as an information processing center. The user device is the initiator of automatic labeling and serves as the initiator of the automatic labeling request, which is usually initiated by the user through the user device.

[0112] The data processing device can be a cloud server, a network server, an application server, a management server, or other devices or servers with data processing functions. The data processing device receives an automatic labeling request from the intelligent terminal through an interaction interface, and performs image processing in the form of machine learning, deep learning, search, reasoning, decision making, etc. through the storage memory and the processor for data processing. The storage memory in the data processing device can be a general term, including local storage and a database for storing historical data. The database can be on the data processing device or on other network servers.

[0113] In Figure 2a the automatic labeling system shown, the user device can receive instructions from the user. For example, the user device can obtain multiple visual data input / selected by the user. Visual data refers to data related to visual information, including image and visual perception data, such as picture data, point cloud data, infrared image data, radar depth data, etc. Then, the user device initiates a request to the data processing device, so that the data processing device performs corresponding image processing applications based on the visual data obtained by the user device, thereby obtaining corresponding processing results for the visual data. For example, the user device can obtain multiple visual data input by the user, and then initiate an image processing request to the data processing device, so that the data processing device processes the visual data based on automatic labeling technology, thereby obtaining processing results corresponding to the visual data, i.e., labeling results.

[0114] In Figure 2a the automatic labeling system shown, the data processing device can execute the multi-modal automatic labeling method of the embodiments of the present application.

[0115] Figure 2b Another structural diagram of the automatic labeling system provided by the embodiments of the present application is shown in Figure 2b In the automatic labeling system shown, the user device directly serves as the data processing device. The user device can directly obtain input from the user and directly process the input by the hardware of the user device itself. The specific process is similar to Figure 2a and can refer to the description above, which will not be repeated here.

[0116] In Figure 2b the automatic labeling system shown, the user device can receive instructions from the user. For example, the user device can obtain multiple visual data selected by the user in the user device, and then perform image processing applications (e.g., automatic labeling, etc.) on the visual data by the user device itself, thereby obtaining corresponding processing results for the multiple visual data.

[0117] In Figure 2b the automatic labeling system shown, the user device can receive instructions from the user. For example, the user device can obtain multiple visual data selected by the user in the user device, and then perform image processing applications (e.g., automatic labeling, etc.) on the visual data by the user device itself, thereby obtaining corresponding processing results for the multiple visual data.

[0118] It can be understood that the automatic labeling system as shown in the above Figure 2a and Figure 2b , can be carried in a cloud platform or independent software facing users, users can upload multiple visual data through the cloud platform, download the automatic labeling result, so as to realize the automatic labeling of visual data; Or through the installation of independent software on the intelligent terminal, the automatic labeling of local visual data. In addition, as shown in the above Figure 2a and Figure 2b , the automatic labeling system can also be carried in an auxiliary labeling system facing an artificial labeling team. The artificial labeling team screens and feeds back the labeling results output by the automatic labeling system to form an active learning process, and finally outputs artificial labeling information based on visual data.

[0119] Figure 3 A schematic diagram of the system 100 architecture provided by the embodiments of the present application is shown in Figure 3 , the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with external devices. The user can input data to the I / O interface 112 through the client device 140, and the input data in the embodiments of the present application can include: various tasks to be scheduled, callable resources and other parameters.

[0120] During the pre-processing of the input data by the execution device 110, or during the execution of the calculation module 111 of the execution device 110 to perform calculations and other related processing (such as the implementation of the functions of the labeling model in the present application), the execution device 110 can call data, codes, etc. in the data storage system 150 for corresponding processing, and can also store the data, instructions, etc. obtained by the corresponding processing in the data storage system 150.

[0121] Finally, the I / O interface 112 returns the processing result to the client device 140, thereby providing it to the user.

[0122] It is worth noting that the training device 120 can generate corresponding target models / rules based on different training data for different targets or different tasks, which can be used to achieve the above-mentioned targets or complete the above-mentioned tasks, thereby providing the required results for the user. Among them, the training data can be stored in the database 130, and comes from the training samples collected by the data acquisition device 160.

[0123] In Figure 3In the case shown in FIG. 1, the user can manually give input data, which can be operated through the interface provided by the I / O interface 112. In another case, the client device 140 can automatically send input data to the I / O interface 112, and if the user needs to authorize the client device 140 to automatically send input data, the user can set the corresponding permission in the client device 140. The user can view the results output by the execution device 110 in the client device 140, and the specific presentation form can be a specific way such as display, sound, action, etc. The client device 140 can also act as a data collection end, collect the input data input into the I / O interface 112 and the output results output by the I / O interface 112 as new sample data, and store them into the database 130. Of course, the collection can also be performed without the client device 140, and the I / O interface 112 can directly store the input data input into the I / O interface 112 and the output results output by the I / O interface 112 as new sample data into the database 130.

[0124] It is worth noting that, Figure 3 The system architecture shown in the figure is only a schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 3 , the data storage system 150 is an external memory relative to the execution device 110, and in other cases, the data storage system 150 can also be placed in the execution device 110. As shown in Figure 3 , the labeled model can be obtained by training the training device 120.

[0125] An embodiment of the present application also provides a chip, which includes a neural network processor NPU. The chip can be arranged in the execution device 110 as shown in Figure 3 , to complete the calculation work of the calculation module 111. The chip can also be arranged in the training device 120 as shown in Figure 3 , to complete the training work of the training device 120 and output the target model / rule.

[0126] The method provided by the present application will be described below from the training side and the application side of the labeled model.

[0127] The multi-modal automatic labeling method provided in the embodiments of the present application relates to processing of image data, and can be applied to methods such as data training, machine learning, and deep learning. The multi-modal input data in the present application is preprocessed, trained, and the like, to obtain a trained labeling model. Furthermore, the multi-modal automatic labeling method provided in the embodiments of the present application can use the trained labeling model to input input data (for example, the multi-modal input data in the present application) into the labeling model to obtain output data (for example, the labeling result in the present application). It should be noted that the training method of the labeling model and the multi-modal automatic labeling method provided in the embodiments of the present application are based on the same concept and can be understood as two parts of a system or two stages of an overall process: for example, a model training stage and a model application stage.

[0128] I. Model application stage

[0129] In the embodiments of the present application, the model application stage describes how the execution device 110 processes input data using the labeling model. For details, please refer to Figure 4 , Figure 4 A flowchart of the multi-modal automatic labeling method provided in the embodiments of the present application is shown in FIG. 1, which is specifically executed by the execution device 110 and includes the following steps. Figure 4

[0130] 401. Obtain first data and second data corresponding to a first time, the first data and the second data being data of different modalities.

[0131] In the embodiments of the present application, feature fusion is performed on the multi-modal input data to realize data labeling. Therefore, multi-modal data needs to be obtained first, and the multi-modal data includes first data and second data, which are data of different modalities. It can be understood that a modality refers to data including multiple types of information, which can come from different sensors, sources, or expression methods, and covers multiple perception modalities. Generally, each modality provides a unique perspective of data, which can enhance the understanding and solution of a certain problem or task.

[0132] ​In the method provided in the embodiments of the present application, the visual data in automatic driving can be specifically annotated, and therefore the multi-modal data can include: picture data captured by a vehicle-mounted camera; laser radar data, specifically three-dimensional point cloud data, sensed and generated by a laser radar sensor; infrared images generated by an infrared sensor by sensing the heat of an object; high-precision map data including map data of roads, traffic signs, lane lines, etc., which can be used for fusion with vehicle sensing data. Correspondingly, in one case, the performing device can be configured with a camera, a laser radar, an infrared sensor, etc., for directly collecting the first data and the second data; in another case, the performing device receives the first data and the second data sent by other electronic devices. It should be understood that the examples herein are only for the convenience of understanding the present solution and are not used to limit the present solution. The first data and the second data can be a combination of any different data above. It should be understood that the first data and the second data are only used to represent that the method in the embodiments of the present application needs to obtain multiple data, i.e., third data, fourth data, etc. at the first time, and the present application does not limit the number of multi-modal data.

[0133] The embodiments of the present application include processing input data at multiple time points, and combining one or more historical time point data in the processing process, so as to obtain the annotation result corresponding to the time point. The first time point can be any time point except the first time point in the foregoing multiple time points.

[0134] In addition, in one possible implementation, after obtaining the multi-modal input data (the first data and the second data), the multi-modal input data can be preprocessed before being used for training or other tasks. One preprocessing method can be to manually define patch or voxel, i.e., to cut or extract small blocks or voxels from the original data. The purpose of this is to obtain smaller and more local data parts for better feature extraction and processing.

[0135] 402, performing attention mechanism-based processing on the first data, the second data, the first historical feature, and the second historical feature to obtain first spatio-temporal features and second spatio-temporal features corresponding to the first time point, wherein the first historical feature and the second historical feature are respectively the first spatio-temporal features and the second spatio-temporal features corresponding to the second time point, the second time point is a time point before the first time point, the second time point is a time point before the first time point, the first spatio-temporal features correspond to the first data, and the second spatio-temporal features correspond to the second data.

[0136] In the embodiments of the present application, after obtaining the first data and the second data of different modalities, the execution device performs attention mechanism-based processing on the first data, the second data, the first historical feature, and the second historical feature. The attention mechanism-based processing is a common technique in deep learning, which is used to enhance the model's attention and processing of important information in the input data. The attention mechanism calculates the correlation weights between different positions or elements in the data, so that the model can pay more attention to the relevant parts during processing and ignore the unimportant parts. After performing the attention mechanism-based processing on the above data, the first spatio-temporal feature and the second spatio-temporal feature corresponding to the first moment are obtained. Among them, the first historical feature and the second historical feature are the first spatio-temporal feature and the second spatio-temporal feature corresponding to the second moment respectively, and the second moment is a moment before the first moment.

[0137] For example, define the first moment as the t moment, where t is an integer greater than 1: the data corresponding to the t moment are the first data D1t, the second data D2t, the first spatio-temporal feature F1t, and the second spatio-temporal feature F2t. At this time, the second moment is a moment before the first moment, specifically including: (1) the second moment is any moment before the first moment, that is, the second moment can be any one or more of 1, 2, 3, …, t-1; (2) the second moment is one moment before the first moment, that is, the second moment is the t-1 moment.

[0138] It can be understood that the attention mechanism-based processing of the input data of each moment can obtain the feature corresponding to the current moment, and in the embodiments of the present application, the processing process also combines historical features, that is, the features corresponding to the moments before the current moment are interactively processed, so that the features obtained in the embodiments achieve cross-time feature interaction. In addition, since the data input in the embodiments of the present application is multi-modal data at the current moment, the multi-modal data is interactively processed in the output feature process, so that the features obtained in the embodiments also achieve cross-space feature interaction. In summary, since the data processed based on the attention mechanism in step 402 includes the interaction between multi-modal data (first data and second data) and the interaction between historical features (first historical features corresponding to the first data and second historical features corresponding to the second data), the features output after this step are spatio-temporal features that include spatial concepts (cross-modal) and temporal concepts (cross-time).

[0139] In a specific implementation method, please refer to Figure 5 , Figure 5 for a possible implementation method of the embodiments corresponding to Figure 4 The flowchart of a possible implementation method of the embodiments corresponding to Figure 5 is shown in FIG. 4, and step 402 specifically includes:

[0140] 4021, respectively, the first data and the second data are processed based on a self-attention mechanism to obtain first spatial features and second spatial features.

[0141] It can be understood that the self-attention mechanism is a special form of attention mechanism for calculating the correlation weight between different positions or elements within the data. In the embodiment, first, the first data and the second data are processed based on a self-attention mechanism, that is, the first data is processed based on a self-attention mechanism to obtain first spatial features; the second data is processed based on a self-attention mechanism to obtain second spatial features. Processing based on a self-attention mechanism is used to capture the relevance between different positions or elements within the data, and these relevance is used to better process the data. Specifically, when the self-attention mechanism is applied to visual image data such as the first data and the second data in the embodiment, the global context and feature relationship of the image are captured by calculating the similarity between different positions in the image. This helps the model better understand the objects and scenes in the image.

[0142] 4022, the first spatial features and the second spatial features are processed based on a cross-attention mechanism to obtain first interaction features and second interaction features, wherein the first interaction features correspond to the first data, and the second interaction features correspond to the second data.

[0143] It can be understood that after obtaining the first spatial features and the second spatial features, the two spatial features are processed based on a cross-attention mechanism to make the two features have a spatial concept, and then the first interaction features and the second interaction features are obtained. In the specific embodiment, it can be understood that the first data and the second data have independent processing procedures respectively, in the processing procedure of the first data, the first interaction features corresponding to the first data are obtained by adding the second spatial features and interacting with the first spatial features; in the processing procedure of the second data, the second interaction features corresponding to the second data are obtained by adding the first spatial features and interacting with the second spatial features.

[0144] 4023, the first interaction features and the first historical features are processed based on a cross-attention mechanism to obtain first spatio-temporal features corresponding to the first time;

[0145] 4024, the second interaction features and the second historical features are processed based on a cross-attention mechanism to obtain second spatio-temporal features corresponding to the first time.

[0146] It can be understood that the interaction between the first spatial features and the second spatial features is spatial interaction, and after spatial interaction, the historical features are interacted to obtain spatio-temporal features.

[0147] In practice, attention mechanisms are often used in conjunction with Transformer models, i.e., with... Figure 5 The cross-modal spatiotemporal Transformer model corresponding to the illustrated method flow can specifically include a self-attention layer, a multimodal cross-attention layer, and a temporal cross-attention layer. Furthermore, during the cross-modal spatiotemporal feature extraction process, a Transformer model can be composed of multiple stacked cross-modal spatiotemporal Transformer blocks, each of which includes a self-attention layer, a multimodal cross-attention layer, and a temporal cross-attention layer.

[0148] For easier understanding, please refer to Figure 6 , Figure 6 This is a schematic diagram of the structure of a cross-modal spatiotemporal Transformer block provided in an embodiment of this application, including: a self-attention layer 601, a multimodal cross-attention layer 602, and a temporal cross-attention layer 603.

[0149] The self-attention layer 601 is used to acquire first data (or second data) and process the first data (or second data) based on the self-attention mechanism to output first spatial features (or second spatial features).

[0150] The multimodal cross-attention layer 602 is used to acquire a first spatial feature (or a second spatial feature) and a second spatial feature (or a first spatial feature), and to process the two features based on a cross-attention mechanism, so that the two features interact with each other and output the first interactive feature (or the second interactive feature).

[0151] The time-cross attention layer 603 is used to obtain a first interaction feature (or a second interaction feature) and a first historical feature (or a second historical feature) corresponding to the first data (or the second data). The first interaction feature (or the second interaction feature) and the first historical feature (or the second historical feature) are processed based on the cross-attention mechanism, so that the two features interact with each other, and the first spatiotemporal feature (or the second spatiotemporal feature) is output.

[0152] In this embodiment, after the multimodal input data (first data and second data) undergoes data preprocessing corresponding to each modality, its cross-modal spatiotemporal features are extracted through multiple cross-modal spatiotemporal Transformer block layers to achieve cross-modal and cross-spatiotemporal feature interaction.

[0153] In one possible implementation, based on Figure 5 The method flow shown may also include enhancements to the output features of each step, specifically including:

[0154] The first spatial feature, the second spatial feature, the first interaction feature, the second interaction feature, the first spatio-temporal feature, or the second spatio-temporal feature is enhanced based on a multi-layer perceptron.

[0155] A multi-layer perceptron (MLP) can perform multi-layer nonlinear transformation on input data, thereby extracting high-level abstract features in the data. Through the superposition of multiple hidden layers, the MLP can learn more complex and abstract representations in the data, which helps better distinguish different categories or perform more complex tasks.

[0156] After the input data is processed based on the self-attention mechanism or the cross-attention mechanism, it can be input into the MLP for enhancement. It can be understood that one or more of the first spatial feature, the second spatial feature, the first interaction feature, the second interaction feature, the first spatio-temporal feature, or the second spatio-temporal feature can be enhanced according to actual needs by those skilled in the art.

[0157] In one possible implementation method, step 4023 specifically includes: performing cross-attention mechanism-based processing on the first spatial feature, the first interaction feature, and the first historical feature to obtain the first spatio-temporal feature corresponding to the first time;

[0158] Step 4024 specifically includes: performing cross-attention mechanism-based processing on the second spatial feature, the second interaction feature, and the second historical feature to obtain the second spatio-temporal feature corresponding to the first time.

[0159] It can be understood that in the Transformer model, skip connections are a technique that directly connects input data to output data. This technique allows information to be directly transmitted across layers in the network, helping to solve the problem of gradient disappearance and promoting the training and performance of the model. Skip connections directly add input data to the output of the Transformer block. In this way, the information of the input data can be directly transmitted to the subsequent layer, so that the network can better capture low-level and high-level features and improve the representation ability of the model. Therefore, through the way of skip connection, the spatial feature after the self-attention layer can be directly transmitted to the subsequent time cross-attention layer, which helps to stabilize the training process and improve the convergence and generalization ability of the model.

[0160] For ease of understanding, please refer to Figure 7 , Figure 7This is a schematic diagram of another cross-modal spatiotemporal Transformer block provided in this application embodiment. Based on the cross-modal spatiotemporal Transformer block shown in Figure 6, in this embodiment, spatial features are first learned through a self-attention layer and a multilayer perceptron; then, cross-modal spatial features are learned through a multimodal cross-attention layer, and the spatial features are enhanced by a perceptron; then, cross-modal interactive features are generated through skip connections; finally, historical features enhanced by the self-attention layer are combined through a temporal cross-attention layer, the features are enhanced by a perceptron, and then spatiotemporal interactive features are generated through skip connections.

[0161] In the process of cross-modal spatiotemporal feature extraction, the Transformer model can be composed of multiple cross-modal spatiotemporal Transformer blocks stacked together. That is, in the embodiments of this application, the Transformer model can be composed of multiple such... Figure 6 The Transformer block shown can be composed of multiple blocks, such as... Figure 7 The Transformer model shown consists of a Transformer block, and can also include, for example, a Transformer block. Figure 6 The Transformer block shown also includes, for example, Figure 7 The Transformer block shown is not limited in this application.

[0162] 403. Perform feature fusion on the first spatiotemporal features and the second spatiotemporal features corresponding to the first time point to obtain the multimodal features corresponding to the first time point.

[0163] It is understandable that for input data of different modalities, after processing based on a self-attention mechanism, corresponding spatiotemporal features are obtained, specifically: a first spatiotemporal feature corresponding to the first data and a second spatiotemporal feature corresponding to the second data. Feature fusion is then performed on the first and second spatiotemporal features to obtain multimodal features.

[0164] In one possible implementation, while fusing the first and second spatiotemporal features, the method also includes acquiring multimodal historical features, which are multimodal features corresponding to the second time step.

[0165] For easier understanding, please refer to Figure 8 , Figure 8 A schematic diagram of the cross-modal spatiotemporal feature fusion layer provided in this application embodiment includes:

[0166] The first multi-modal cross-attention layer 801 and the second cross-modal attention layer 802. It can be understood that the number of multi-modal cross-attention layers should be consistent with the number of input data. When the input data is the first data and the second data, the multi-modal cross-attention layer includes the first multi-modal cross-attention layer 801 corresponding to the first data and the second cross-modal attention layer 802 corresponding to the second data; when the input data further includes third data, the third data is different modal data from the first data and the second data, and the cross-modal spatio-temporal feature fusion layer further includes a third multi-modal cross-attention layer.

[0167] After obtaining the multi-modal historical data, the multiple spatio-temporal features corresponding to the multi-modal data are sequentially processed based on the cross-attention mechanism, and the processing result is further processed with the next spatio-temporal feature based on the cross-attention mechanism. Finally, the multi-modal feature corresponding to the first time is obtained. The multi-modal feature also combines historical data, and the finally output multi-modal feature has time characteristics.

[0168] 404, performing labeling processing based on the multi-modal feature, and outputting a labeling result corresponding to the first time.

[0169] It can be understood that the fused unified multi-modal feature can be used as input to the labeling task for labeling processing, thereby obtaining a labeling result corresponding to the first time.

[0170] In one possible implementation method, the same multi-modal feature can be labeled by different labeling models. At this time, step 404 specifically includes:

[0171] 4041, respectively performing labeling processing on the multi-modal feature by a first labeling model and a second labeling model to obtain corresponding first labeling information and second labeling information, the first labeling model and the second labeling model being different labeling models;

[0172] 4042, performing information interaction on the first labeling information and the second labeling information, and outputting a labeling result.

[0173] In the embodiment of the application, the fused unified multi-modal feature representation is input to the multi-task labeling module (first labeling model and second labeling model) for labeling processing, and the model output information of each is interacted to obtain the final result. In the embodiment of the application, the multi-modal data is input to different labeling models to obtain labeling information, and then the labeling information is interacted to obtain the final labeling result, thereby improving the accuracy of the output labeling result of the automatic labeling system. In one possible implementation method, the labeling model includes a three-dimensional (3D) target frame detection model, a semantic segmentation model, a target tracking model, a behavior analysis model, etc., which are not exhaustively listed here.

[0174] In one possible implementation method, step 4042 specifically includes:

[0175] The second annotation information is input into the first annotation model as prior information of the multi-modal feature to obtain a first annotation result.

[0176] The first annotation information is input into the second annotation model as prior information of the multi-modal feature to obtain a second annotation result.

[0177] The annotation result is output, and the annotation result includes the first annotation result and the second annotation result.

[0178] In the embodiments of the present application, the first annotation information and the second annotation information are interacted, specifically, the output results of different annotation task modules are interacted as prior information of other annotation tasks.

[0179] For better understanding, please refer to Figure 9 , Figure 9 The architecture diagram of the multi-modal automatic annotation system provided by the embodiments of the present application.

[0180] The specific process of the system is described as follows:

[0181] 1) After the input data of the multi-modal (modal one, modal two) is preprocessed by each modal data, the system extracts the cross-modal spatio-temporal features thereof through multiple cross-modal spatio-temporal Transformer block layers to realize the cross-modal and cross-spatio-temporal feature interaction. The data preprocessing can be a method of manually defining the extraction of data patches or voxels. The basic composition of the cross-modal spatio-temporal Transformer block layer includes a self-attention layer, a multi-modal cross-attention layer, and a time cross-attention layer. In the cross-modal spatio-temporal feature extraction process, the model can be stacked by L cross-modal spatio-temporal Transformer block layers, where L is an integer greater than or equal to 1;

[0182] 2) For the features extracted by different modalities, the system generates a unified multi-modal feature representation through a cross-modal spatio-temporal feature fusion layer. The cross-modal spatio-temporal feature fusion layer is composed of multiple multi-modal cross-attention layers, and the number of multi-modal cross-attention layers is consistent with the number of input modalities;

[0183] 3) For the unified multi-modal feature representation, the system inputs it into the multi-task (task one, task two) annotation module for annotation processing, and the respective input and output are interacted as prior information of other tasks. The annotation task module can be composed of a convolutional neural network or a fully connected layer neural network, and according to the specific task requirement, the annotation task module can be a three-dimensional target frame detection layer, a scene pixel semantic segmentation layer, etc.

[0184] 4) The system sets the outputs of different annotation task modules as multi-task annotation information as output.

[0185] 5) In addition, the system can use the multi-task annotation information as feedback information as the true value label of the unlabeled multi-modal data to form new labeled input data and promote the iterative optimization training of the system.

[0186] In another possible implementation method, step 4042 specifically includes:

[0187] generating prompt information according to the first annotation information and the second annotation information through the prompt information encoder;

[0188] inputting the prompt information into the first annotation model and the second annotation model respectively to obtain corresponding first annotation results and second annotation results;

[0189] outputting the annotation results, the annotation results including the first annotation results and the second annotation results.

[0190] It can be understood that the first annotation information and the second annotation information can also be input into the prompt information encoder prompt as cross-task automatic prompt information to learn prompt embedding features, generate prompt information, and then input the prompt information into the multi-task annotation module to refine the annotation results of each module to obtain corresponding annotation results.

[0191] For ease of understanding, please refer to Figure 10 and Figure 11 , Figure 10 the structural diagram of the multi-task annotation module provided by the embodiments of the present application, Figure 11 the architecture diagram of the multi-modal automatic annotation system based on the multi-task annotation module shown in Figure 10 .

[0192] It can be understood that Figure 10 the task annotation model adopted by the module shown in is taken as an example of a three-dimensional target frame model and a scene pixel semantic segmentation model. Specifically as follows:

[0193] After obtaining the fused unified multi-modal features through steps 401 to 403 in the above method embodiments, the features are input into the multi-task (task one, task two) annotation module for annotation processing, and then the annotation information of each module is input into the prompt information encoder as cross-task automatic prompt information to learn prompt embedding features and input into the multi-task annotation module to refine the annotation results of each module. Specifically, the multi-task annotation module is constructed by a coarse-to-fine strategy, specifically:

[0194] (1) The three-dimensional target box detection model inputs the multi-modal features into a three-dimensional target detector to generate an initial three-dimensional target detection box, and then generates three-dimensional target box labeling information results through a size refinement layer, a position and direction refinement layer, and a tracking consistency constraint layer.

[0195] (2) The scene pixel semantic segmentation model inputs the multi-modal features and the intermediate features of each modality into a modality decoder, predicts the scene semantic labeling information of each modality through a prediction head, and finally collects the multi-modal scene semantic labeling information.

[0196] The prompt information encoder is composed of a convolutional neural network or a fully connected layer neural network. In addition to the cross-task labeling information as the automatic prompt information, it can also be a manually designed prompt information, which is used to assist the refinement of multi-task labeling information.

[0197] The multi-modal automatic labeling method provided by the embodiments of the present application can use the cross-modal spatio-temporal Transformer block layer and the cross-modal spatio-temporal fusion layer to construct the spatio-temporal complementary information of multi-modal data, generate fused unified features, and realize the output of automatic labeling results of multiple functions, thereby improving the accuracy of the automatic labeling system. The Transformer block layer fuses the cross-modal information and the cross-spatio-temporal information of the multi-modal input through a cross-attention mechanism, and integrates them into each basic block to establish the cross-modal spatio-temporal interaction of multi-modal features. The multi-task consistent information collaborative optimization module optimizes the labeling information using a coarse-to-fine strategy, and converts the multi-task output information or the manually designed information into prompt embedding features through a prompt encoder and fuses them into the multi-modal features, realizes multi-task consistent information collaborative learning, and provides an automatic labeling method based on multi-task prompt.

[0198] It can be understood that, in addition to being used for automatic labeling of automatic driving data, the technical solution of the present application can also be used in other technical fields after modification of the multi-task output module. Specifically, the Transformer model constructed based on the cross-modal spatio-temporal Transformer block layer in the present application is widely applicable to processing various types of input data, such as automatic driving multi-modal data, human-computer interaction multi-modal data, visual natural language multi-modal data, etc. Its role is to extract and fuse multi-modal feature representations for subsequent specific task processing. Therefore, by modifying the multi-task output module, the technical solution can be applied to automatic driving, human-computer interaction, image-text processing and other technical fields based on the Transformer model designed in the present application. The main change of the technical solution is the modification of the multi-task output module.

[0199] II. Model training phase:

[0200] In the embodiment of the present application, the training stage describes the process of how the training device 120 generates a mature labeling model using the training data set in the database 130. For details, please refer to Figure 12 , Figure 12 A flowchart of the training method of the labeling model provided in the embodiment of the present application, the training method of the labeling model provided in the embodiment of the present application is executed by the training device and can include the following steps:

[0201] 1201, obtaining first data and second data corresponding to a first time, the first data and the second data being data of different modalities.

[0202] 1202, processing the first data and the second data through a to-be-trained model to obtain a labeling result corresponding to the first time, wherein the to-be-trained model is configured to: perform attention mechanism-based processing on the first data, the second data, a first historical feature, and a second historical feature to obtain a first spatio-temporal feature and a second spatio-temporal feature corresponding to the first time, the first historical feature and the second historical feature being the first spatio-temporal feature and the second spatio-temporal feature corresponding to a second time respectively, the second time being a time before the first time; perform feature fusion on the first spatio-temporal feature and the second spatio-temporal feature to obtain a multi-modal feature; and perform labeling processing based on the multi-modal feature to output the labeling result corresponding to the first time.

[0203] 1203, training the to-be-trained model based on the labeling result to obtain a target model.

[0204] In the embodiment, the training device specifies the specific implementation of steps 1201 to 1203, which is similar to the specific implementation of steps 401 to 404 executed by the execution device in the corresponding embodiment. For details, please refer to the understanding, which will not be repeated here. Figure 4 The specific implementation of steps 401 to 404 executed by the execution device in the corresponding embodiment is similar, which can be understood by referring to the understanding, and will not be repeated here.

[0205] The training device repeatedly executes steps 1201 to 1203 to realize iterative training of the target model until a preset condition is met, and obtains a trained target model.

[0206] In one possible implementation method, the attention mechanism-based processing on the first data, the second data, the first historical feature, and the second historical feature to obtain the first spatio-temporal feature and the second spatio-temporal feature corresponding to the first time specifically includes:

[0207] respectively performing self-attention mechanism-based processing on the first data and the second data to obtain a first spatial feature and a second spatial feature;

[0208] The first spatial feature and the second spatial feature are subjected to attention mechanism-based interaction processing to obtain a first interaction feature and a second interaction feature, wherein the first interaction feature corresponds to the first data, and the second interaction feature corresponds to the second data;

[0209] The first interaction feature and the first historical feature are subjected to cross-attention mechanism-based processing to obtain a first spatio-temporal feature corresponding to the first time point;

[0210] The second interaction feature and the second historical feature are subjected to cross-attention mechanism-based processing to obtain a second spatio-temporal feature corresponding to the first time point.

[0211] In a possible implementation method, at least one of the first spatial feature, the second spatial feature, the first interaction feature, the second interaction feature, the first spatio-temporal feature, or the second spatio-temporal feature is enhanced based on a multi-layer perceptron.

[0212] In a possible implementation method, the first interaction feature and the first historical feature are subjected to cross-attention mechanism-based processing to obtain a first spatio-temporal feature corresponding to the first time point; and the second interaction feature and the second historical feature are subjected to cross-attention mechanism-based processing to obtain a second spatio-temporal feature corresponding to the first time point, specifically including:

[0213] The first spatial feature, the first interaction feature, and the first historical feature are subjected to cross-attention mechanism-based processing to obtain a first spatio-temporal feature corresponding to the first time point;

[0214] The second spatial feature, the second interaction feature, and the second historical feature are subjected to cross-attention mechanism-based processing to obtain a second spatio-temporal feature corresponding to the first time point.

[0215] In a possible implementation method, the multi-modal feature is subjected to annotation processing to output an annotation result, specifically including:

[0216] The multi-modal feature is subjected to annotation processing by a first annotation model and a second annotation model respectively to obtain corresponding first annotation information and second annotation information, the first annotation model and the second annotation model being different annotation models;

[0217] The first annotation information and the second annotation information are subjected to information interaction to output an annotation result.

[0218] In a possible implementation method, the first annotation information and the second annotation information are subjected to information interaction to output an annotation result, specifically including:

[0219] The second annotation information is input into the first annotation model as prior information of the multi-modal feature to obtain a first annotation result;

[0220] The first annotation information is input into the second annotation model as prior information of the multi-modal feature to obtain a second annotation result.

[0221] The annotation result includes the first annotation result and the second annotation result.

[0222] In a possible implementation method, the first annotation information and the second annotation information are interacted to output the annotation result, and the method specifically includes:

[0223] The prompt information is generated by the prompt information encoder according to the first annotation information and the second annotation information;

[0224] The prompt information is input into the first annotation model and the second annotation model to obtain the corresponding first annotation result and second annotation result.

[0225] The annotation result includes the first annotation result and the second annotation result.

[0226] In the above Figures 1 to 12 Based on the embodiments corresponding to the above-mentioned scheme of the present application, the following provides a related device for implementing the above-mentioned scheme.

[0227] For details Figure 13 , Figure 13 A structural diagram of a multi-modal automatic annotation device provided by the embodiments of the present application, the multi-modal automatic annotation device 1300 includes:

[0228] The input layer 1301 is configured to obtain first data and second data corresponding to a first time, the first data and the second data being data of different modalities;

[0229] The multi-modal spatio-temporal attention layer 1302 is configured to perform attention mechanism-based processing on the first data, the second data, a first historical feature, and a second historical feature to obtain a first spatio-temporal feature and a second spatio-temporal feature corresponding to the first time, wherein the first historical feature and the second historical feature are respectively a first spatio-temporal feature and a second spatio-temporal feature corresponding to a second time, the second time being a time before the first time, the first spatio-temporal feature corresponding to the first data, and the second spatio-temporal feature corresponding to the second data;

[0230] The spatio-temporal feature fusion layer 1303 is configured to perform feature fusion on the first spatio-temporal feature and the second spatio-temporal feature corresponding to the first time to obtain a multi-modal feature corresponding to the first time.

[0231] The task annotation layer 1304 is configured to perform annotation processing based on the multi-modal feature to output an annotation result corresponding to the first time.

[0232] In a possible implementation, the multi-modal space-time attention layer 1302 is specifically configured to perform self-attention mechanism-based processing on the first data and the second data respectively to obtain first spatial features and second spatial features; perform attention mechanism-based interaction processing on the first spatial features and the second spatial features to obtain first interaction features and second interaction features, wherein the first interaction features correspond to the first data, and the second interaction features correspond to the second data; perform cross-attention mechanism-based processing on the first interaction features and the first historical features to obtain first space-time features corresponding to the first moment; and perform cross-attention mechanism-based processing on the second interaction features and the second historical features to obtain second space-time features corresponding to the first moment.

[0233] In a possible implementation, the multi-modal space-time attention layer 1302 is specifically configured to perform self-attention mechanism-based processing on the first data and the second data respectively to obtain first spatial features and second spatial features; perform attention mechanism-based interaction processing on the first spatial features and the second spatial features to obtain first interaction features and second interaction features, wherein the first interaction features correspond to the first data, and the second interaction features correspond to the second data; perform cross-attention mechanism-based processing on the first interaction features and the first historical features to obtain first space-time features corresponding to the first moment; and perform cross-attention mechanism-based processing on the second interaction features and the second historical features to obtain second space-time features corresponding to the first moment.

[0234] In a possible implementation, the multi-modal space-time attention layer 1302 is specifically configured to perform self-attention mechanism-based processing on the first data and the second data respectively to obtain first spatial features and second spatial features; perform attention mechanism-based interaction processing on the first spatial features and the second spatial features to obtain first interaction features and second interaction features, wherein the first interaction features correspond to the first data, and the second interaction features correspond to the second data; perform cross-attention mechanism-based processing on the first interaction features and the first historical features to obtain first space-time features corresponding to the first moment; and perform cross-attention mechanism-based processing on the second interaction features and the second historical features to obtain second space-time features corresponding to the first moment.

[0235] In a possible implementation, the multi-modal space-time attention layer 1302 is specifically configured to perform self-attention mechanism-based processing on the first data and the second data respectively to obtain first spatial features and second spatial features; perform attention mechanism-based interaction processing on the first spatial features and the second spatial features to obtain first interaction features and second interaction features, wherein the first interaction features correspond to the first data, and the second interaction features correspond to the second data; perform cross-attention mechanism-based processing on the first interaction features and the first historical features to obtain first space-time features corresponding to the first moment; and perform cross-attention mechanism-based processing on the second interaction features and the second historical features to obtain second space-time features corresponding to the first moment.

[0236] In a possible implementation, the multi-modal space-time attention layer 1302 is specifically configured to perform self-attention mechanism-based processing on the first data and the second data respectively to obtain first spatial features and second spatial features; perform attention mechanism-based interaction processing on the first spatial features and the second spatial features to obtain first interaction features and second interaction features, wherein the first interaction features correspond to the first data, and the second interaction features correspond to the second data; perform cross-attention mechanism-based processing on the first interaction features and the first historical features to obtain first space-time features corresponding to the first moment; and perform cross-attention mechanism-based processing on the second interaction features and the second historical features to obtain second space-time features corresponding to the first moment.

[0237] The present embodiment also provides a training device of an annotation model, please refer to Figure 14 , Figure 14 The training device of the annotation model provided in the present embodiment has a structure diagram, and the training device 1400 of the annotation model includes:

[0238] The obtaining module 1401 is configured to obtain first data and second data corresponding to the first moment, the first data and the second data being data of different modalities.

[0239] The generating module 1402 is configured to obtain the labeling result corresponding to the first time by processing the first data and the second data by using a to-be-trained model. The to-be-trained model is configured to: perform attention mechanism-based processing on the first data, the second data, the first historical feature, and the second historical feature to obtain first space-time features and second space-time features corresponding to the first time, the first historical feature and the second historical feature being the first space-time features and the second space-time features corresponding to a second time, the second time being a time before the first time, the first space-time features corresponding to the first data, and the second space-time features corresponding to the second data; perform feature fusion on the first space-time features and the second space-time features corresponding to the first time to obtain multi-modal features; and perform labeling processing based on the multi-modal features to output the labeling result corresponding to the first time.

[0240] The training module 1403 is configured to train the to-be-trained model based on the labeling result to obtain a target model.

[0241] Next, an execution device provided by an embodiment of the present application is introduced. Please refer to Figure 15 , Figure 15 FIG. 1 is a structural schematic diagram of an execution device provided by an embodiment of the present application. The execution device 1500 can be embodied as an automatic driving vehicle, a mobile phone, a tablet computer, a notebook computer, a desktop computer, a monitoring data processing device, etc., and is not limited herein. The execution device 1500 can be deployed with the multi-modal automatic labeling apparatus 1300 described in the corresponding embodiments. Figure 13 The execution device 1500 can implement the functions of the execution device in the corresponding embodiments. Figures 3 to 11 Specifically, the execution device 1500 includes a receiver 1501, a transmitter 1502, a processor 1503, and a memory 1504 (wherein the number of processors 1503 in the execution device 1500 can be one or more, and one processor is taken as an example in the embodiment). Figure 15 The processor 1503 can include an application processor 15031 and a communication processor 15032. In some embodiments of the present application, the receiver 1501, the transmitter 1502, the processor 1503, and the memory 1504 can be connected by a bus or other means.

[0242] The memory 1504 can include a read-only memory and a random access memory, and provide the processor 1503 with instructions and data. A part of the memory 1504 can also include a non-volatile random access memory (NVRAM). The memory 1504 stores processor and operation instructions, executable modules or data structures, or subsets thereof, or expanded sets thereof, wherein the operation instructions can include various operation instructions for implementing various operations.

[0243] The processor 1503 controls the operation of the execution device. In a specific application, various components of the execution device are coupled together by a bus system, which can include a data bus, a power bus, a control bus, and a state signal bus, etc. However, for the sake of clarity, various buses are referred to as a bus system in the figure.

[0244] The method disclosed in the embodiments of the present application can be applied to or implemented by the processor 1503. The processor 1503 can be an integrated circuit chip with a signal processing capability. In the implementation process, the steps of the above method can be completed by hardware integrated logic circuits in the processor 1503 or by instructions in the form of software. The processor 1503 described above can be a general processor, a digital signal processor (DSP), a microprocessor, or a microcontroller. It can further include an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The processor 1503 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general processor can be a microprocessor or any conventional processor. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, or other mature storage media in the art. The storage medium is located in the storage 1504, and the processor 1503 reads the information in the storage 1504 and combines the hardware to complete the steps of the above method.

[0245] The receiver 1501 can be used to receive input digital or character information, and generate signal input related to the relevant settings and function control of the execution device. The transmitter 1502 can be used to output digital or character information through the first interface; the transmitter 1502 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 1502 can also include a display device such as a display screen.

[0246] In the embodiments of the present application, the processor 1503 is configured to perform Figures 3 to 11 The processor 1503 in the corresponding embodiment performs the multi-modal automatic labeling method executed by the execution device. The application processor 15031 in the processor 1503 performs the specific manner of each step described above, and the details are described in the embodiments of the present application. Figures 3 to 11Corresponding to each method embodiment is based on the same concept, the technical effect brought by which is same as that in the present application Figures 3 to 11 Corresponding to each method embodiment is same, and specific content can be referred to the description in the method embodiments shown in the foregoing of the present application, which will not be repeated here.

[0247] The present application also provides a training device, please refer to Figure 16 , Figure 16 is a structural schematic diagram of the training device provided by the present application, and the training device 1600 can be deployed with Figure 14 The training device 1400 described in the corresponding embodiment is used to realize Figure 12 The function of the training device in the corresponding embodiment, specifically, the training device 1600 is realized by one or more servers, and the training device 1600 can be compared greatly due to different configurations or performances, and can include one or more central processing units (CPU) 1622 (for example, one or more processors) and a memory 1632, one or more storage media 1630 (for example, one or more mass storage devices) for storing application programs 1642 or data 1644. Among them, the memory 1632 and the storage medium 1630 can be temporary storage or persistent storage. The program stored in the storage medium 1630 can include one or more modules (not shown in the figure), each module can include a series of instruction operations in the training device. Further, the central processing unit 1622 can be arranged to communicate with the storage medium 1630, and execute a series of instruction operations in the storage medium 1630 on the training device 1600.

[0248] The training device 1600 can also include one or more power supplies 1626, one or more wired or wireless network interfaces 1650, one or more input and output interfaces 1658, and / or one or more operating systems 1641, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0249] In the present application, the central processing unit 1622 is used to execute Figure 12 The multi-modal automatic labeling method executed by the training device in the corresponding embodiment. The central processing unit 1622 executes the specific mode of each step, which is same as that in the present application Figure 12 Corresponding to each method embodiment is based on the same concept, the technical effect brought by which is same as that in the present application Figure 12 Corresponding to each method embodiment is same, and specific content can be referred to the description in the method embodiments shown in the foregoing of the present application, which will not be repeated here.

[0250] The embodiment of the present application further provides a computer program product, when running on a computer, causes the computer to perform the steps executed by the execution device in the method described in the foregoing Figures 3 to 11 The embodiment of the present application further provides a computer program product, when running on a computer, causes the computer to perform the steps executed by the training device in the method described in the foregoing Figure 12 The embodiment of the present application further provides a computer program product, when running on a computer, causes the computer to perform the steps executed by the training device in the method described in the foregoing

[0251] The embodiment of the present application further provides a computer readable storage medium, the computer readable storage medium stores a program for signal processing, when running on a computer, causes the computer to perform the steps executed by the execution device in the method described in the foregoing Figures 3 to 11 The embodiment of the present application further provides a computer readable storage medium, the computer readable storage medium stores a program for signal processing, when running on a computer, causes the computer to perform the steps executed by the execution device in the method described in the foregoing Figure 12 The embodiment of the present application further provides a computer readable storage medium, the computer readable storage medium stores a program for signal processing, when running on a computer, causes the computer to perform the steps executed by the training device in the method described in the foregoing

[0252] The environment data processing apparatus, the neural network training apparatus, the execution device and the training device provided by the embodiment of the present application can be a chip, which comprises a processing unit and a communication unit. The processing unit can be a processor, and the communication unit can be an input / output interface, a pin or a circuit, etc. The processing unit can execute computer execution instructions stored in a storage unit, so as to make the chip execute the environment data processing method described in the foregoing Figures 3 to 11 The processing unit can execute computer execution instructions stored in a storage unit, so as to make the chip execute the neural network training method described in the foregoing Figure 12 The storage unit can be a storage unit in the chip, such as a register, a cache, etc. The storage unit can also be a storage unit outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0253] In addition, it should be noted that the apparatus embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., they can be located in one place, or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment. In addition, in the apparatus embodiment provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines.

[0254] Those skilled in the art can clearly understand that the application can be implemented by means of software plus necessary universal hardware, and of course can also be implemented by means of dedicated hardware including special integrated circuit, special CPU, special memory, special component, etc. Generally, any function completed by computer program can be easily implemented by corresponding hardware, and the specific hardware structure for implementing the same function can also be various, such as analog circuit, digital circuit or special circuit, etc. However, for the application, software program implementation is a better embodiment. Based on such understanding, the technical solution of the application or the part of the application which makes contribution to the prior art can be embodied in the form of software product, which is stored in readable storage medium, such as computer floppy disk, U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a plurality of instructions for making a computer device (which can be personal computer, training device or network device, etc.) execute the method described in various embodiments of the application.

[0255] In the above embodiments, the implementation can be achieved by software, hardware, firmware or any combination thereof, entirely or partially. When implemented by software, the implementation can be achieved in the form of a computer program product, entirely or partially.

[0256] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the flow or function described in the embodiments of the application is generated entirely or partially. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another, for example, the computer instructions can be transmitted from one website, computer, training device or data center to another website, computer, training device or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be stored by a computer or a data storage device such as a training device, a data center, etc. integrated with one or more available media sets. The available medium can be a magnetic medium (such as a floppy disk, a hard disk, a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

Claims

1. A multi-modal automatic annotation method, characterized by, The method comprises: obtaining first data and second data corresponding to a first time, the first data and the second data being data of different modalities; performing attention mechanism-based processing on the first data, the second data, a first historical feature, and a second historical feature to obtain a first spatio-temporal feature and a second spatio-temporal feature corresponding to the first time, wherein the first historical feature and the second historical feature are respectively a first spatio-temporal feature and a second spatio-temporal feature corresponding to a second time, the second time being a time prior to the first time, the first spatio-temporal feature corresponding to the first data, and the second spatio-temporal feature corresponding to the second data; the attention mechanism-based processing on the first data, the second data, the first historical feature, and the second historical feature to obtain the first spatio-temporal feature and the second spatio-temporal feature corresponding to the first time specifically comprises: performing self-attention mechanism-based processing on the first data and the second data respectively to obtain a first spatial feature and a second spatial feature; performing attention mechanism-based interaction processing on the first spatial feature and the second spatial feature to obtain a first interaction feature and a second interaction feature, wherein the first interaction feature corresponds to the first data, and the second interaction feature corresponds to the second data; performing cross-attention mechanism-based processing on the first interaction feature and the first historical feature to obtain the first spatio-temporal feature corresponding to the first time; and performing cross-attention mechanism-based processing on the second interaction feature and the second historical feature to obtain the second spatio-temporal feature corresponding to the first time; performing feature fusion on the first spatio-temporal feature and the second spatio-temporal feature corresponding to the first time to obtain a multi-modal feature corresponding to the first time; performing annotation processing based on the multi-modal feature to output an annotation result corresponding to the first time.

2. The method of claim 1, wherein, Further comprising: enhancing at least one of the first spatial feature, the second spatial feature, the first interaction feature, the second interaction feature, the first spatio-temporal feature, or the second spatio-temporal feature based on a multi-layer perceptron.

3. The method according to claim 1 or 2, characterized in that, the cross-attention mechanism-based processing on the first interaction feature and the first historical feature to obtain the first spatio-temporal feature corresponding to the first time; the cross-attention mechanism-based processing on the second interaction feature and the second historical feature to obtain the second spatio-temporal feature corresponding to the first time, specifically comprising: the cross-attention mechanism-based processing on the first spatial feature, the first interaction feature, and the first historical feature to obtain the first spatio-temporal feature corresponding to the first time; the cross-attention mechanism-based processing on the second spatial feature, the second interaction feature, and the second historical feature to obtain the second spatio-temporal feature corresponding to the first time.

4. The method of claim 1, wherein, the annotation processing based on the multi-modal feature to output the annotation result, specifically comprising: The multi-modal feature is labeled by a first labeling model and a second labeling model respectively to obtain corresponding first labeling information and second labeling information, the first labeling model and the second labeling model being different labeling models; The first labeling information and the second labeling information are interacted to output a labeling result.

5. The method of claim 4, wherein, The first labeling information and the second labeling information are interacted to output a labeling result, specifically including: The second labeling information is input into the first labeling model as prior information of the multi-modal feature to obtain a first labeling result; The first labeling information is input into the second labeling model as prior information of the multi-modal feature to obtain a second labeling result; A labeling result is output, and the labeling result includes the first labeling result and the second labeling result.

6. The method of claim 4, wherein, The first labeling information and the second labeling information are interacted to output a labeling result, specifically including: Prompt information is generated according to the first labeling information and the second labeling information by a prompt information encoder; The prompt information is input into the first labeling model and the second labeling model respectively to obtain corresponding first labeling results and second labeling results; A labeling result is output, and the labeling result includes the first labeling result and the second labeling result.

7. The method according to any one of claims 4 to 6, characterized in that, The labeling model includes a three-dimensional (3D) target box detection model, a semantic segmentation model, a target tracking model, and a behavior analysis model. 8.A method for training a labeling model, the method comprising: including: First data and second data corresponding to a first time are obtained, the first data and the second data being data of different modalities; obtaining a label result corresponding to the first moment by processing the first data and the second data through a to-be-trained model, wherein the to-be-trained model is configured to: perform attention mechanism-based processing on the first data, the second data, a first historical feature, and a second historical feature to obtain a first spatio-temporal feature and a second spatio-temporal feature corresponding to the first moment, the first historical feature and the second historical feature being a first spatio-temporal feature and a second spatio-temporal feature corresponding to a second moment respectively, the second moment being a moment before the first moment, the first spatio-temporal feature corresponding to the first data, and the second spatio-temporal feature corresponding to the second data; performing feature fusion on the first spatio-temporal feature and the second spatio-temporal feature corresponding to the first moment to obtain a multi-modal feature; performing labeling processing based on the multi-modal feature to output a label result corresponding to the first moment; and the attention mechanism-based processing on the first data, the second data, the first historical feature, and the second historical feature to obtain the first spatio-temporal feature and the second spatio-temporal feature corresponding to the first moment specifically includes: performing self-attention mechanism-based processing on the first data and the second data respectively to obtain a first spatial feature and a second spatial feature; performing attention mechanism-based interaction processing on the first spatial feature and the second spatial feature to obtain a first interaction feature and a second interaction feature, wherein the first interaction feature corresponds to the first data, and the second interaction feature corresponds to the second data; performing cross-attention mechanism-based processing on the first interaction feature and the first historical feature to obtain the first spatio-temporal feature corresponding to the first moment; and performing cross-attention mechanism-based processing on the second interaction feature and the second historical feature to obtain the second spatio-temporal feature corresponding to the first moment; training the to-be-trained model based on the label result to obtain a target model.

9. The method of claim 8, wherein, Further comprising: enhancing at least one of the first spatial feature, the second spatial feature, the first interaction feature, the second interaction feature, the first spatio-temporal feature, or the second spatio-temporal feature based on a multi-layer perceptron.

10. The method according to claim 8 or 9, characterized in that, the cross-attention mechanism-based processing on the first interaction feature and the first historical feature to obtain the first spatio-temporal feature corresponding to the first moment; the cross-attention mechanism-based processing on the second interaction feature and the second historical feature to obtain the second spatio-temporal feature corresponding to the first moment, specifically including: the cross-attention mechanism-based processing on the first spatial feature, the first interaction feature, and the first historical feature to obtain the first spatio-temporal feature corresponding to the first moment; the cross-attention mechanism-based processing on the second spatial feature, the second interaction feature, and the second historical feature to obtain the second spatio-temporal feature corresponding to the first moment.

11. A multi-modal automatic annotation apparatus, characterized by, comprising: an input layer configured to obtain first data and second data corresponding to a first moment, the first data and the second data being data of different modalities; The multi-modal spatio-temporal attention layer is specifically configured to perform self-attention mechanism-based processing on the first data and the second data respectively to obtain first spatial features and second spatial features; perform attention mechanism-based interaction processing on the first spatial features and the second spatial features to obtain first interaction features and second interaction features, wherein the first interaction features correspond to the first data, and the second interaction features correspond to the second data; perform cross-attention mechanism-based processing on the first interaction features and the first historical features to obtain the first spatio-temporal features corresponding to the first moment; and perform cross-attention mechanism-based processing on the second interaction features and the second historical features to obtain the second spatio-temporal features corresponding to the first moment. The multi-modal spatio-temporal attention layer is specifically configured to perform self-attention mechanism-based processing on the first data and the second data respectively to obtain first spatial features and second spatial features; perform attention mechanism-based interaction processing on the first spatial features and the second spatial features to obtain first interaction features and second interaction features, wherein the first interaction features correspond to the first data, and the second interaction features correspond to the second data; perform cross-attention mechanism-based processing on the first interaction features and the first historical features to obtain the first spatio-temporal features corresponding to the first moment; and perform cross-attention mechanism-based processing on the second interaction features and the second historical features to obtain the second spatio-temporal features corresponding to the first moment. The spatio-temporal feature fusion layer is configured to perform feature fusion on the first spatio-temporal features and the second spatio-temporal features corresponding to the first moment to obtain multi-modal features corresponding to the first moment. The task labeling layer is configured to perform labeling processing based on the multi-modal features to output a labeling result corresponding to the first moment.

12. The apparatus of claim 11, wherein The task labeling layer is specifically configured to perform labeling processing on the multi-modal features by using a first labeling model and a second labeling model respectively to obtain corresponding first labeling information and second labeling information, the first labeling model and the second labeling model being different labeling models; and perform information interaction on the first labeling information and the second labeling information to output a labeling result.

13. A training apparatus of an annotation model, characterized by, The apparatus comprises: An acquisition module configured to acquire first data and second data corresponding to a first moment, the first data and the second data being data of different modalities. The generating module is configured to process the first data and the second data by using a to-be-trained model to obtain a labeling result corresponding to the first moment, wherein the to-be-trained model is configured to: perform attention mechanism-based processing on the first data, the second data, a first historical feature, and a second historical feature to obtain a first spatio-temporal feature and a second spatio-temporal feature corresponding to the first moment, the first historical feature and the second historical feature are respectively a first spatio-temporal feature and a second spatio-temporal feature corresponding to a second moment, the second moment is a moment before the first moment, the first spatio-temporal feature corresponds to the first data, and the second spatio-temporal feature corresponds to the second data; perform feature fusion on the first spatio-temporal feature and the second spatio-temporal feature corresponding to the first moment to obtain a multi-modal feature; and perform labeling processing based on the multi-modal feature to output the labeling result corresponding to the first moment; and the attention mechanism-based processing on the first data, the second data, the first historical feature, and the second historical feature to obtain the first spatio-temporal feature and the second spatio-temporal feature corresponding to the first moment specifically includes: performing self-attention mechanism-based processing on the first data and the second data respectively to obtain a first spatial feature and a second spatial feature; performing attention mechanism-based interaction processing on the first spatial feature and the second spatial feature to obtain a first interaction feature and a second interaction feature, wherein the first interaction feature corresponds to the first data, and the second interaction feature corresponds to the second data; and performing cross-attention mechanism-based processing on the first interaction feature and the first historical feature to obtain the first spatio-temporal feature corresponding to the first moment; and performing cross-attention mechanism-based processing on the second interaction feature and the second historical feature to obtain the second spatio-temporal feature corresponding to the first moment. The training module is configured to train the to-be-trained model based on the labeling result to obtain a target model.

14. A computer program product, characterised in that, The computer program, when executed on a computer, causes the computer to perform the method of any one of claims 1 to 7, or causes the computer to perform the method of any one of claims 8 to 10.

15. A computer-readable storage medium, characterized in that, The program, when executed on a computer, causes the computer to perform the method of any one of claims 1 to 7, or causes the computer to perform the method of any one of claims 8 to 10.

16. An execution device, comprising: The execution device comprises a processor and a memory, The memory is configured to store a program; The processor is configured to execute the program in the memory, so that the execution device performs the method of any one of claims 1 to 7.

17. A training apparatus, characterized by The training device comprises a processor and a memory, The memory is configured to store a program; The processor is configured to execute the program in the memory, so that the training device performs the method of any one of claims 8 to 10.

Images