Model training method, data processing method, device, equipment, medium and product

Abstract

The invention discloses a model training method and device, a data processing method and device, equipment, a medium and a product. The method comprises the following steps: training a first feature extraction network and a feature analysis network in an original model based on at least one first training sample to obtain a primary model; training a first feature extraction network, a second feature extraction network and a feature analysis network in the primary model based on at least one second training sample to obtain an intermediate model; and training the second feature extraction network and the feature analysis network in the intermediate model based on at least one third training sample to obtain a target model. Through the technical scheme of the invention, the data annotation cost and training cost of the model can be reduced.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer technology, and in particular to a model training method, data processing method, apparatus, device, medium and product. Background Technology

[0002] End-to-end technology is a unified framework or approach designed to integrate multiple tasks. Its aim is to handle different tasks through a single model.

[0003] In practical production and daily life, joint training methods using joint annotation are generally used to train models. However, joint training of models using joint annotation has several problems: end-to-end systems require a large amount of labeled data for training, and data carrying labeled information is very expensive. Therefore, training models using jointly labeled data is costly and requires significant training resources. Summary of the Invention

[0004] This invention provides a model training method, data processing method, apparatus, device, medium, and product that can reduce the data annotation cost and training cost of models.

[0005] According to one aspect of the present invention, a model training method is provided, the model comprising: a first feature extraction network, a second feature extraction network, and a feature parsing network, the method comprising:

[0006] The first feature extraction network and the feature parsing network in the original model are trained based on at least one first training sample to obtain a primary model. The first training sample includes first input modality data and the annotation information corresponding to the first input modality data.

[0007] The first feature extraction network, the second feature extraction network, and the feature parsing network in the primary model are trained based on at least one second training sample to obtain an intermediate model. The second training sample includes second input modal data and the annotation information corresponding to the second input modal data.

[0008] The second feature extraction network and the feature parsing network in the intermediate model are trained based on at least one third training sample to obtain the target model. The third training sample includes third input modality data and the annotation information corresponding to the third input modality data.

[0009] In some embodiments of the present invention, the feature parsing network includes: multiple task learning modules, which include a current target task learning module and a key task learning module, and the union of the current target task learning module and the key task learning module constitutes multiple task learning modules.

[0010] In some embodiments of the present invention, a primary model is obtained by training a first feature extraction network and a feature parsing network in the original model based on at least one first training sample, including:

[0011] Based on at least one first training sample, the first feature extraction network associated with the at least one first training sample and the current target task learning module in the original model are trained to obtain a primary model; wherein, the first input modal data includes at least one input modal data corresponding to the current target task learning module, and the annotation information corresponding to the first input modal data includes at least one annotation information corresponding to the current target task learning module.

[0012] In some embodiments of the present invention, the second feature extraction network and the feature parsing network in the intermediate model are trained based on at least one third training sample to obtain the target model, including:

[0013] While training the intermediate model based on at least one third training sample, the training parameters of the first feature extraction network in the intermediate model are locked to adjust the training parameters of the second feature extraction network and the key task learning module in the intermediate model to obtain the target model. The third input modal data includes at least one input modal data corresponding to the key task learning module, and the annotation information corresponding to the third input modal data includes at least one annotation information corresponding to the key task learning module.

[0014] In some embodiments of the present invention, the second input modal data includes all types of modal data required for the target model to make predictions; the first input modal data includes at least one type of modal data required for the target model to make predictions; the third input modal data includes at least one type of modal data required for the target model to make predictions; the types of modal data included in the first input modal data are different from those included in the third input modal data; the union of the modal data included in the first input modal data and the modal data included in the third input modal data represents all types of modal data required for the target model to make predictions; the annotation information corresponding to the second input modal data includes all types of output results of the target model; the annotation information corresponding to the first input modal data includes at least one type of output result of the target model; the annotation information corresponding to the third input modal data includes at least one type of output result of the target model; the types of output results included in the annotation information corresponding to the first input modal data are different from those included in the annotation information corresponding to the third input modal data; the union of the output results included in the annotation information corresponding to the first input modal data and the output results included in the annotation information corresponding to the third input modal data represents all types of output results of the target model.

[0015] In some embodiments of the present invention, the first input modal data includes: environmental perception samples, which include: frame samples and / or point cloud samples; the annotation information corresponding to the first input modal data includes: a first obstacle label, a second obstacle label, and a road structure label; the second input modal data includes: driving-related samples, which include: sensor samples and navigation planning samples; the sensor samples include: environmental perception samples and state samples; the annotation information corresponding to the second input modal data includes: a first obstacle label, a second obstacle label, a road structure label, an obstacle prediction trajectory, decision information, and a driving trajectory; the third input modal data includes: state samples and navigation planning samples; the state samples include: position samples and attitude samples; the annotation information corresponding to the third input modal data includes: obstacle prediction trajectory, decision information, and a driving trajectory.

[0016] In some embodiments of the present invention, the first feature extraction network includes a backbone network, the second feature extraction network includes an encoder, and the feature parsing network includes a decoder.

[0017] According to another aspect of the present invention, a data processing method is provided, the data processing method comprising:

[0018] Acquire target multimodal data, wherein the target multimodal data includes modal data of all modal types required for the target model to make predictions;

[0019] The target multimodal data is input into the target model to obtain the target output result set.

[0020] In some embodiments of the present invention, the target multimodal data includes: driving-related data of the current vehicle, or driving-related data of the intelligent robot. The driving-related data includes: sensor information and navigation planning information. The sensor information includes: environmental perception information and state information. The environmental perception information includes: current frame data and / or current point cloud data. The target output result set includes: first obstacle information, second obstacle information, target road structure, target obstacle trajectory, target decision information, and target driving trajectory.

[0021] According to another aspect of the present invention, a model training apparatus is provided, the model comprising: a first feature extraction network, a second feature extraction network, and a feature parsing network, the apparatus comprising:

[0022] The first training module is used to train the first feature extraction network and the feature parsing network in the original model based on at least one first training sample to obtain a primary model. The first training sample includes first input modality data and the annotation information corresponding to the first input modality data.

[0023] The second training module is used to train the first feature extraction network, the second feature extraction network, and the feature parsing network in the primary model based on at least one second training sample to obtain an intermediate model. The second training sample includes second input modal data and the annotation information corresponding to the second input modal data.

[0024] The third training module is used to train the second feature extraction network and the feature parsing network in the intermediate model based on at least one third training sample to obtain the target model. The third training sample includes third input modality data and the annotation information corresponding to the third input modality data.

[0025] In some embodiments of the present invention, the feature parsing network includes: multiple task learning modules, which include a current target task learning module and a key task learning module, and the union of the current target task learning module and the key task learning module constitutes multiple task learning modules.

[0026] In some embodiments of the present invention, the first training module is specifically used for:

[0027] Based on at least one first training sample, the first feature extraction network associated with the at least one first training sample and the current target task learning module in the original model are trained to obtain a primary model; wherein, the first input modal data includes at least one input modal data corresponding to the current target task learning module, and the annotation information corresponding to the first input modal data includes at least one annotation information corresponding to the current target task learning module.

[0028] In some embodiments of the present invention, the third training module is specifically used for:

[0029] While training the intermediate model based on at least one third training sample, the training parameters of the first feature extraction network in the intermediate model are locked to adjust the training parameters of the second feature extraction network and the key task learning module in the intermediate model to obtain the target model. The third input modal data includes at least one input modal data corresponding to the key task learning module, and the annotation information corresponding to the third input modal data includes at least one annotation information corresponding to the key task learning module.

[0030] In some embodiments of the present invention, the second input modal data includes all types of modal data required for the target model to make predictions; the first input modal data includes at least one type of modal data required for the target model to make predictions; the third input modal data includes at least one type of modal data required for the target model to make predictions; the types of modal data included in the first input modal data are different from those included in the third input modal data; the union of the modal data included in the first input modal data and the modal data included in the third input modal data represents all types of modal data required for the target model to make predictions; the annotation information corresponding to the second input modal data includes all types of output results of the target model; the annotation information corresponding to the first input modal data includes at least one type of output result of the target model; the annotation information corresponding to the third input modal data includes at least one type of output result of the target model; the types of output results included in the annotation information corresponding to the first input modal data are different from those included in the annotation information corresponding to the third input modal data; the union of the output results included in the annotation information corresponding to the first input modal data and the output results included in the annotation information corresponding to the third input modal data represents all types of output results of the target model.

[0031] In some embodiments of the present invention, the first input modal data includes: environmental perception samples, which include: frame samples and / or point cloud samples; the annotation information corresponding to the first input modal data includes: a first obstacle label, a second obstacle label, and a road structure label; the second input modal data includes: driving-related samples, which include: sensor samples and navigation planning samples; the sensor samples include: environmental perception samples and state samples; the annotation information corresponding to the second input modal data includes: a first obstacle label, a second obstacle label, a road structure label, an obstacle prediction trajectory, decision information, and a driving trajectory; the third input modal data includes: state samples and navigation planning samples; the state samples include: position samples and attitude samples; the annotation information corresponding to the third input modal data includes: obstacle prediction trajectory, decision information, and a driving trajectory.

[0032] In some embodiments of the present invention, the first feature extraction network includes a backbone network, the second feature extraction network includes an encoder, and the feature parsing network includes a decoder.

[0033] According to another aspect of the present invention, a data processing apparatus is provided, the apparatus comprising:

[0034] The target multimodal data acquisition module is used to acquire target multimodal data, wherein the target multimodal data includes modal data of all modal types required for the target model to make predictions;

[0035] The data processing module is used to input the target multimodal data into the target model to obtain the target output result set.

[0036] In some embodiments of the present invention, the target multimodal data includes: driving-related data of the current vehicle, or driving-related data of the intelligent robot. The driving-related data includes: sensor information and navigation planning information. The sensor information includes: environmental perception information and state information. The environmental perception information includes: current frame data and / or current point cloud data. The target output result set includes: first obstacle information, second obstacle information, target road structure, target obstacle trajectory, target decision information, and target driving trajectory.

[0037] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0038] At least one processor; and

[0039] A memory communicatively connected to the at least one processor; wherein,

[0040] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the model training method or data processing method described in any embodiment of the present invention.

[0041] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions, the computer instructions being configured to cause a processor to execute and implement the model training method or data processing method described in any embodiment of the present invention.

[0042] According to another aspect of the present invention, a computer program product is provided, which, when executed by a processor, implements a model training method or a data processing method as described in any embodiment of the present invention.

[0043] The technical solution of this embodiment divides the model training phase into three stages: The first stage involves training the first feature extraction network and the feature parsing network in the original model based on at least one first training sample to obtain a primary model; the second stage involves training the first feature extraction network, the second feature extraction network, and the feature parsing network in the primary model based on at least one second training sample to obtain an intermediate model; and the third stage involves training the second feature extraction network and the feature parsing network in the intermediate model based on at least one third training sample to obtain the target model. This allows for the use of different labeled data at different stages based on the data annotation situation, reducing the data annotation cost and training cost of the model while accelerating the model convergence speed.

[0044] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0045] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a flowchart of the first model training method in this embodiment of the invention;

[0047] Figure 2 This is a flowchart of the second model training method in this embodiment of the invention;

[0048] Figure 3 This is a schematic diagram of the first stage of model training in an embodiment of the present invention;

[0049] Figure 4 This is a schematic diagram of the second stage of model training in an embodiment of the present invention;

[0050] Figure 5 This is a schematic diagram of the third stage model training in an embodiment of the present invention;

[0051] Figure 6 This is a flowchart of the first data processing method in this embodiment of the invention;

[0052] Figure 7 This is a flowchart of the second data processing method in this embodiment of the invention;

[0053] Figure 8 This is a schematic diagram of the structure of a model training device according to an embodiment of the present invention;

[0054] Figure 9 This is a schematic diagram of the structure of a data processing device according to an embodiment of the present invention;

[0055] Figure 10 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0056] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0057] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0058] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0059] Example 1

[0060] Figure 1 This is a flowchart of a first model training method provided in an embodiment of the present invention. This embodiment is applicable to model training. The method can be executed by the model training device in this embodiment, which can be implemented in software and / or hardware, such as... Figure 1 As shown, the method specifically includes the following steps:

[0061] S101. Train the first feature extraction network and the feature parsing network in the original model based on at least one first training sample to obtain the primary model.

[0062] In this embodiment, the first training sample includes first input modal data and corresponding annotation information. The first input modal data contains at least one type of modal data required for the target model to make predictions, and the corresponding annotation information contains at least one type of output result from the target model. It should be noted that the type of modal data included in the first input modal data is related to the type of output result contained in the corresponding annotation information. For example, if the output results of type A and type B depend only on modal data of type C and type D, then the first input modal data includes: modal data of type C and modal data of type D, and the corresponding annotation information includes: annotation information corresponding to the output results of type A and annotation information corresponding to the output results of type B. It should also be noted that the first training sample can be used as training samples for the first feature extraction network and the feature parsing network.

[0063] In this embodiment, the first feature extraction network can be a backbone network, and the feature parsing network can be a decoder.

[0064] In this embodiment, the original model includes: a first feature extraction network, a second feature extraction network, and a feature parsing network. The primary model includes: a first feature extraction network, a second feature extraction network, and a feature parsing network.

[0065] In this embodiment, the method of training the first feature extraction network and the feature parsing network in the original model based on at least one first training sample to obtain the primary model can be as follows: input the first input modality data into the original model to obtain the first prediction result; determine the difference between the first prediction result and the annotation information corresponding to the first input modality data, and train the first feature extraction network and the feature parsing network in the original model based on the above difference to obtain the primary model.

[0066] It should be noted that the initial model is obtained by training the first feature extraction network and the feature parsing network in the original model based on at least one first training sample. Since the first input modal data contains fewer types of modal data and the corresponding annotation information contains fewer types of output results, the cost of obtaining the first training sample is lower compared to the full range of modal data and output results required for the target model to make predictions.

[0067] S102. Train the first feature extraction network, the second feature extraction network, and the feature parsing network in the primary model based on at least one second training sample to obtain the intermediate model.

[0068] In this embodiment, the second training sample includes second input modal data and corresponding annotation information. The second input modal data contains all types of modal data required for the target model to make predictions. The annotation information corresponding to the second input modal data contains all types of output results of the target model.

[0069] In this embodiment, the second feature extraction network can be an encoder. The intermediate model includes: a first feature extraction network, a second feature extraction network, and a feature parsing network.

[0070] In this embodiment, the intermediate model is obtained by training the first feature extraction network, the second feature extraction network, and the feature parsing network in the primary model based on at least one second training sample. The intermediate model is obtained by: inputting the second input modality data into the primary model to obtain the second prediction result; determining the difference between the second prediction result and the annotation information corresponding to the second input modality data; and training the first feature extraction network, the second feature extraction network, and the feature parsing network in the primary model based on the above difference to obtain the intermediate model.

[0071] It should be noted that training the first feature extraction network, the second feature extraction network, and the feature parsing network in the primary model based on at least one second training sample can fully integrate the information between the first feature extraction network, the second feature extraction network, and the feature parsing network, thereby reducing the loss of information transmission.

[0072] In this embodiment, although training the first feature extraction network, the second feature extraction network, and the feature parsing network in the primary model based on at least one second training sample is a holistic training, the number of training samples required for holistic training in this embodiment is much smaller than that required for joint training compared to joint training. Furthermore, since the training is based on the primary model obtained by training the first feature extraction network and the feature parsing network in the original model based on the first training sample, the training speed is improved compared to joint training.

[0073] S103. Train the second feature extraction network and the feature parsing network in the intermediate model based on at least one third training sample to obtain the target model.

[0074] In this embodiment, the third training sample includes third input modal data and corresponding annotation information. The third input modal data contains at least one type of modal data required for prediction by the target model, and the corresponding annotation information contains at least one type of output result from the target model. The modal data contained in the first input modal data is of a different type than the modal data contained in the third input modal data, and the union of the modal data contained in the first and third input modal data represents all types of modal data required for prediction by the target model. The output results contained in the annotation information corresponding to the first input modal data are of a different type than the output results contained in the annotation information corresponding to the third input modal data, and the union of the output results contained in the annotation information corresponding to the first and third input modal data represents all types of output results from the target model.

[0075] In this embodiment, the output result of the target type can be pre-selected, and then the third training sample can be determined based on the output result of the pre-selected target type.

[0076] It's important to note that dividing model training into three stages, using different labeled data at each stage based on the data annotation situation, reduces the need for joint labeled data and accelerates model convergence. Furthermore, the intermediate model trained in the second stage can be saved as the base model for the third stage. If adjustments are needed to the output of the third stage, there's no need to retrain the first and second stages; training can proceed directly from the pre-saved intermediate model, saving significant training resources.

[0077] In this embodiment, the dependencies between all types of modal data and all types of output results required for the target model to make predictions are pre-defined (the dependencies between modal data and output results can be reflected by the correlation coefficient between them; for example, if the correlation coefficient between modal data and output results is greater than a threshold, then a dependency between them is determined). Based on these dependencies, the model training is segmented, and training samples are generated for each training stage. For example, if the modal data required for the target model to make predictions includes five types of modal data (modal data 1, modal data 2, modal data 3, modal data 4, and modal data 5) and the output results of the target model include six types of output results (output result 1, output result 2, output result 3, output result 4, output result 5, and output result 6), then the model training is segmented, and training samples are generated for each training stage. Output results 1, 2, and 3 are pre-defined and are dependent on modal data 1 and 2. Output results 4, 5, and 6 are dependent on modal data 3, 4, and 5. The first training sample includes the first input modal data and the corresponding annotation information. The first input modal data includes the input modal sample corresponding to modal data 1 and the input modal sample corresponding to modal data 2. The annotation information corresponding to the first input modal data includes the annotation information corresponding to output results 1, output results 2, and output results 3. The second training samples include the second input modal data and the corresponding annotation information. The second input modal data includes: the input modal sample corresponding to modal data 1, the input modal sample corresponding to modal data 2, the input modal sample corresponding to modal data 3, the input modal sample corresponding to modal data 4, and the input modal sample corresponding to modal data 5. The annotation information corresponding to the second input modal data includes: the annotation information corresponding to output result 1, the annotation information corresponding to output result 2, the annotation information corresponding to output result 3, the annotation information corresponding to output result 4, the annotation information corresponding to output result 5, and the annotation information corresponding to output result 6. The third training samples include the third input modal data and the corresponding annotation information. The third input modal data includes: the input modal sample corresponding to modal data 3, the input modal sample corresponding to modal data 4, and the input modal sample corresponding to modal data 5. The annotation information corresponding to the third input modal data includes: the annotation information corresponding to output result 4, the annotation information corresponding to output result 5, and the annotation information corresponding to output result 6.

[0078] It should be noted that the annotation information in the first, second, and third training samples can be obtained by either manual annotation or by using corresponding annotation models. For example, the first input modality data can be input into the first annotation model to obtain the annotation information corresponding to the first input modality data; the second input modality data can be input into the second annotation model to obtain the annotation information corresponding to the second input modality data; and the third input modality data can be input into the third annotation model to obtain the annotation information corresponding to the third input modality data. All three annotation models are pre-trained models.

[0079] Optionally, the first input modal data includes: environmental perception samples, which include: frame samples and / or point cloud samples; the annotation information corresponding to the first input modal data includes: a first obstacle label, a second obstacle label, and a road structure label; the second input modal data includes: driving-related samples, which include: sensor samples and navigation planning samples; the sensor samples include: environmental perception samples and state samples; the annotation information corresponding to the second input modal data includes: a first obstacle label, a second obstacle label, a road structure label, an obstacle prediction trajectory, decision information, and a driving trajectory; the third input modal data includes: state samples and navigation planning samples, which include: position samples and attitude samples; the annotation information corresponding to the third input modal data includes: obstacle prediction trajectory, decision information, and a driving trajectory.

[0080] In this embodiment, the obstacle curve is the predicted obstacle trajectory. Decision information is information determined based on the predicted obstacle trajectory, and may include lane changing, deceleration, etc. For example, if there is an obstacle ahead, the decision information may include any one of: decelerating to yield, changing lanes to bypass, or changing lanes to overtake. The driving trajectory is a planned trajectory. The planned trajectory can be generated based on the decision information; for example, if the decision information is to change lanes to the right, then a trajectory for changing lanes to the right will be planned.

[0081] In this embodiment, the first obstacle label can be information about obstacles of a fixed shape. For example, the fixed-shape obstacle could be a pedestrian, bicycle, car, etc. The second obstacle label can be information about obstacles of a non-fixed shape. For example, the non-fixed-shape obstacle could be an excavator, fence, crane, streetlight, etc. The second obstacle label can be OCC (Occupancy Network) information. It enables accurate identification and segmentation of obstacles in complex road environments, thereby improving the safety and flexibility of intelligent driving systems. OCC information can be a three-dimensional occupancy grid representing the spatial distribution of obstacles in an image.

[0082] In a specific example, model training is divided into three stages. The first stage involves inputting environmental perception samples into the original model to obtain first predicted obstacle information x1, second predicted obstacle information x2, and predicted road structure information x3. The differences between the first predicted obstacle information x1 and the first obstacle label, the second predicted obstacle information x2 and the second obstacle label, and the predicted road structure information x3 and the road structure label are determined. Based on these differences, the first feature extraction network and the feature parsing network in the original model are trained to obtain a primary model. The second stage involves inputting environmental perception samples, state samples, and navigation planning samples into the primary model to obtain the first predicted obstacle information x4, the second predicted obstacle information x5, the predicted road structure information x6, the predicted obstacle curve x7, the predicted decision information x8, and the predicted driving trajectory x9. Based on the differences between the first predicted obstacle information x4 and the first obstacle label, the second predicted obstacle information x5 and the second obstacle label, the predicted road structure information x6 and the road structure label, the predicted obstacle curve x7 and the obstacle curve, the predicted decision information x8 and the decision information, and the predicted driving trajectory x9 and the driving trajectory, the primary model is trained as a whole to obtain the intermediate model. The third stage involves inputting state samples and navigation planning samples into the intermediate model to obtain predicted obstacle curves x10, predicted decision information x11, and predicted driving trajectory x12. The training parameters of the first feature extraction network in the intermediate model are then locked. Based on the differences between the predicted obstacle curve x10 and the obstacle curve, the differences between the predicted decision information x11 and the decision information, and the differences between the predicted driving trajectory x12 and the driving trajectory, the training parameters of the second feature extraction network and the key task learning module in the intermediate model are adjusted to obtain the target model.

[0083] Optionally, the first feature extraction network includes a backbone network, the second feature extraction network includes an encoder, and the feature parsing network includes a decoder.

[0084] In this embodiment, the backbone network and decoder in the original model are trained based on at least one first training sample to obtain a primary model; the backbone network, encoder, and decoder in the primary model are trained based on at least one second training sample to obtain an intermediate model; and the encoder and decoder in the intermediate model are trained based on at least one third training sample to obtain a target model.

[0085] The technical solution of this embodiment divides the model training phase into three stages: The first stage involves training the first feature extraction network and the feature parsing network in the original model based on at least one first training sample to obtain a primary model; the second stage involves training the first feature extraction network, the second feature extraction network, and the feature parsing network in the primary model based on at least one second training sample to obtain an intermediate model; and the third stage involves training the second feature extraction network and the feature parsing network in the intermediate model based on at least one third training sample to obtain the target model. This approach allows for the use of different labeled data at different stages based on the data annotation situation, reducing the data annotation cost and training cost of the model while accelerating model convergence.

[0086] Example 2

[0087] Figure 2 This is a flowchart of the second model training method in this embodiment of the invention, which is an optimization based on the above embodiment. In this embodiment, the feature parsing network includes: multiple task learning modules, including a current target task learning module and a key task learning module, the union of the current target task learning module and the key task learning module is multiple task learning modules, and a primary model is obtained by training a first feature extraction network associated with at least one first training sample and the current target task learning module based on at least one first training sample; wherein, the first input modal data includes at least one input modal data corresponding to the current target task learning module, and the annotation information corresponding to the first input modal data includes at least one annotation information corresponding to the current target task learning module; the primary model is trained by training a first feature extraction network, a second feature extraction network, and a feature parsing network based on at least one second training sample to obtain an intermediate model; while training the intermediate model based on at least one third training sample, the training parameters of the first feature extraction network in the intermediate model are locked to adjust the training parameters of the second feature extraction network and the key task learning module in the intermediate model to obtain a target model. Figure 2 As shown, the method specifically includes the following steps:

[0088] S201. Based on at least one first training sample, train the first feature extraction network associated with the at least one first training sample and the current target task learning module in the original model to obtain a primary model.

[0089] In this embodiment, the feature parsing network includes multiple task learning modules, which include a current target task learning module and a key task learning module. The union of the current target task learning module and the key task learning module constitutes the multiple task learning modules. It should be noted that the intersection of the current target task learning module and the key task learning module can be empty or non-empty.

[0090] In a specific example, if all task learning modules include: Task Learning Module 1, Task Learning Module 2, Task Learning Module 3, Task Learning Module 4, Task Learning Module 5, and Task Learning Module 6, select a portion of these modules as the current target task learning modules and the remaining portion as key learning modules. Modules 1, 2, and 3 are designated as the current target task learning modules, while Modules 4, 5, and 6 are designated as key task learning modules (the intersection of the current target and key task learning modules is empty). Alternatively, Modules 1, 2, and 3 can be designated as the current target task learning modules, while Modules 3, 4, 5, and 6 are designated as key task learning modules (the intersection of the current target and key task learning modules is not empty).

[0091] In this embodiment, the first input modal data includes at least one input modal data corresponding to the current target task learning module, and the annotation information corresponding to the first input modal data includes at least one annotation information corresponding to the current target task learning module.

[0092] In this embodiment, the first input modality data from the first training sample is input into the original model to obtain the first prediction result corresponding to the current target task learning module; the difference between the first prediction result corresponding to the current target task learning module and at least one labeled information corresponding to the current target task learning module is determined, and the first feature extraction network associated with the at least one first training sample and the current target task learning module are trained according to the difference to obtain a primary model.

[0093] The technical solution of this embodiment divides the feature parsing network into multiple task learning modules, which enables more targeted training. If fine-tuning is required for the target task, it is not necessary to retrain other tasks unrelated to the target task, thus saving a lot of training resources.

[0094] The technical solution of this embodiment uses a subset of task learning modules as the current target task learning module. Based on at least one first training sample, the first feature extraction network associated with the at least one first training sample in the original model and the current target task learning module are trained to obtain a preliminary model. Since the first training sample contains fewer types of modal data and fewer types of output results, the cost of obtaining the first training sample is lower compared to obtaining all types of modal data and all types of output results required for the target model to make predictions. Furthermore, pre-training a subset of task learning modules can improve model training efficiency.

[0095] S202. The primary model is trained as a whole based on at least one second training sample corresponding to all task learning modules.

[0096] In this embodiment, the second input modal data includes multiple input modal data corresponding to all task learning modules, and the annotation information corresponding to the second input modal data includes the annotation information corresponding to all task learning modules.

[0097] In this embodiment, the second input modal data from the second training sample is input into the primary model to obtain the second prediction result; the difference between the second prediction result and the annotation information corresponding to all task learning modules is determined, and the primary model is trained as a whole based on the difference to obtain the intermediate model.

[0098] In this embodiment, the input modal data in the second training sample includes all types of input modal data, and the annotation information includes the annotation information corresponding to all types of output results. Compared with the previous stage, the input modal data and annotation information in the training sample are more comprehensive. Through this stage, the information between the first feature extraction network, the second feature extraction network, and the feature parsing network can be fully integrated, enabling a full representation of the scene during the learning process and reducing information transmission loss in multimodal and multi-task models.

[0099] S203. Using at least one task learning module as a key task learning module, while training the intermediate model based on at least one third training sample, the training parameters of the first feature extraction network in the intermediate model are locked, so as to adjust the training parameters of the second feature extraction network and the key task learning module in the intermediate model to obtain the target model.

[0100] In this embodiment, the third input modal data includes at least one input modal data corresponding to the key task learning module, and the annotation information corresponding to the third input modal data includes at least one annotation information corresponding to the key task learning module.

[0101] In this embodiment, the output of the critical task learning module is the final output, while the output of the current target task learning module can be either the final output or an intermediate result. The output of the current target task learning module is a deterministic result (e.g., obstacle location information, obstacle size, lane line information, etc.), while the output of the critical task learning module is a predicted result (e.g., predicted obstacle trajectory, decision information, predicted vehicle or intelligent robot trajectory, etc.).

[0102] In this embodiment, at least one task learning module is pre-selected as the key task learning module. The third input modality data from the third training sample is input into the intermediate model to obtain the third prediction result. The difference between the third prediction result and at least one labeled information corresponding to the key task learning module is determined. The training parameters of the first feature extraction network in the intermediate model are locked. The training parameters of the second feature extraction network and the key task learning module in the intermediate model are adjusted according to the difference to obtain the target model.

[0103] In this embodiment, the union of the types of annotation information corresponding to the current target task learning module and the types of annotation information corresponding to the key task learning module constitutes all output result types of the target model. For example, the output results of the target model may include: first type information, second type information, third type information, fourth type information, fifth type information, and sixth type information. The annotation information corresponding to the current target task learning module includes first type information, second type information, and third type information, while the annotation information corresponding to the key task learning module includes: annotation information corresponding to third type information, fourth type information, fifth type information, and sixth type information.

[0104] It should be noted that when the union of the annotation information types corresponding to the current target task learning module and the annotation information types corresponding to the key task learning module equals all output result types of the target model, and when the annotation information types corresponding to the current target task learning module and the key task learning module are different, training speed can be improved and training costs reduced. For example, the annotation information corresponding to the current target task learning module may include annotation information corresponding to the first type of information, the second type of information, and the third type of information, while the annotation information corresponding to the key task learning module may include annotation information corresponding to the fourth type of information, the fifth type of information, and the sixth type of information.

[0105] In this embodiment, after the second stage, the information in the network has been fully integrated. At this point, it is necessary to fine-tune the key task learning module, and the output result corresponding to the key task learning module is the final output result. During the training process of this stage, locking the training parameters of the first feature extraction network in the intermediate model is to reduce the influence of the output result of the key task learning module on the output result of the current target task learning module.

[0106] The technical solution in this embodiment divides model training into three stages. Different labeled data are used in different stages based on the data annotation situation, reducing the need for joint labeled data and accelerating model convergence. Simultaneously, the intermediate model trained in the second stage can be saved as the base model for the third stage. If the output result corresponding to the third stage needs adjustment, there is no need to retrain the first and second stages; training can be performed directly based on the pre-saved intermediate model, saving significant training resources.

[0107] It should be noted that if, after training the first feature extraction network and the current target task learning module in the original model based on at least one first training sample, the training parameters of the second feature extraction network and the key task learning module are adjusted in advance based on at least one third training sample before overall training, the accuracy of the output result corresponding to the current target task learning module will be affected during the adjustment of the training parameters of the second feature extraction network and the key task learning module based on at least one third training sample. The technical solution of this embodiment, by performing overall training in advance after training the first feature extraction network and the current target task learning module, and then adjusting the training parameters of the second feature extraction network and the key task learning module, can mitigate the impact on the current target task learning module during the adjustment of the training parameters of the second feature extraction network and the key task learning module, thereby improving the accuracy of model prediction.

[0108] Optionally, the second input modal data includes all types of modal data required for the target model to make predictions; the first input modal data includes at least one type of modal data required for the target model to make predictions; the third input modal data includes at least one type of modal data required for the target model to make predictions; the types of modal data included in the first input modal data and the types of modal data included in the third input modal data are different; the union of the modal data included in the first input modal data and the types of modal data included in the third input modal data represents all types of modal data required for the target model to make predictions; the annotation information corresponding to the second input modal data includes all types of output results of the target model; the annotation information corresponding to the first input modal data includes at least one type of output result of the target model; the annotation information corresponding to the third input modal data includes at least one type of output result of the target model; the types of output results included in the annotation information corresponding to the first input modal data and the types of output results included in the annotation information corresponding to the third input modal data are different; the union of the output results included in the annotation information corresponding to the first input modal data and the types of output results included in the annotation information corresponding to the third input modal data represents all types of output results of the target model.

[0109] In this embodiment, the target model requires M types of modal data for prediction, and the target model includes N types of output results. The second input modal data contains M types of modal data, and the annotation information corresponding to the second input modal data contains N types of output results. The first input modal data contains X types of modal data, where X is less than M, and the annotation information corresponding to the first input modal data contains Y types of output results, where Y is less than N. The third input modal data contains R types of modal data, where R = MX, or R includes at least one type of X in addition to MX, and the annotation information corresponding to the third input modal data contains T types of output results, where T = NY, or T includes at least one type of Y in addition to NY.

[0110] It should be noted that the modal data contained in the first input modal data are of different types than those contained in the third input modal data. The union of the modal data contained in the first and third input modal data represents all types of modal data required for prediction by the target model. Similarly, the output results contained in the annotation information corresponding to the first and third input modal data are of different types. The union of the output results contained in the annotation information corresponding to the first and third input modal data represents all types of output results of the target model. Using training samples that meet the above conditions for segmented training, since multiple task learning modules are trained without repetitive training, the annotation cost can be reduced and the model training speed improved while ensuring model accuracy.

[0111] In a specific example, if the original model is a multi-task end-to-end model, the multi-task end-to-end model requires five types of modal data for prediction: modal data 1, modal data 2, modal data 3, modal data 4, and modal data 5. The multi-task end-to-end model outputs six results corresponding to tasks: the output results for task 1, task 2, task 3, task 4, task 5, and task 6. It should be noted that since tasks 1, 2, and 3 depend on modal data 1 and modal data 2, and the first feature extraction network and feature parsing network are related to modal data 1 and modal data 2, the first input modal data in the first training sample includes two types of modal data: modal data 1 and modal data 2. The annotation information corresponding to the first input modal data includes the annotation information corresponding to the output results of task 1, task 2, and task 3. In the first stage of model training, the first feature extraction network and the feature parsing network in the original model are trained based on at least one first training sample to obtain a primary model. The second input modal data in the second training sample includes five types of modal data: modal data 1, modal data 2, modal data 3, modal data 4, and modal data 5. The annotation information corresponding to the second input modal data includes the annotation information corresponding to the output results of task 1, task 2, task 3, task 4, task 5, and task 6. In the second stage of model training, the first feature extraction network, the second feature extraction network, and the feature parsing network in the primary model are trained based on at least one second training sample to obtain an intermediate model. Tasks 4, 5, and 6 are pre-selected as the core target tasks. The third input modal data in the third training samples includes three types of modal data: modal data 3, modal data 4, and modal data 5. The annotation information corresponding to the third input modal data includes the annotation information corresponding to the output results of task 4, task 5, and task 6. In the third stage of model training, the second feature extraction network and the feature parsing network in the intermediate model are trained based on at least one third training sample to obtain the target model.

[0112] Optionally, the first feature extraction network includes a backbone network, the second feature extraction network includes an encoder, and the feature parsing network includes a decoder.

[0113] In this embodiment, the decoder includes multiple task learning modules.

[0114] In a specific example, training an end-to-end model can be equivalent to training a model with a multimodal input and multiple task outputs. Since different tasks depend on different input modalities of data, the model training can be divided into stages based on the dependencies between modalities and tasks. For example, because the perception task depends on images and point clouds, and the planning task depends on all input modalities of data, the model training can be divided into three stages, such as... Figure 3 As described above, since Task 1, Task 2, and Task 3 depend on Modal Data 1 and Modal Data 2, and the backbone network and decoder are related to Modal Data 1 and Modal Data 2, the first input modal data in the first training sample includes two types of modal data, namely Modal Data 1 and Modal Data 2. The annotation information corresponding to the first input modal data includes: the annotation information corresponding to the output result of Task 1, the annotation information corresponding to the output result of Task 2, and the annotation information corresponding to the output result of Task 3. The first stage of model training ( Figure 3 (The gray area) is used to train the backbone network and decoder in the original model based on at least one first training sample, resulting in a primary model. For example... Figure 4 As shown, the second input modal data in the second training sample includes five types of modal data: modal data 1, modal data 2, modal data 3, modal data 4, and modal data 5. The annotation information corresponding to the second input modal data includes: the annotation information corresponding to the output results of task 1, task 2, task 3, task 4, task 5, and task 6. The second stage of model training ( Figure 4 (The gray portion) is used to train the backbone network, encoder, and decoder in the primary model based on at least one second training sample, resulting in an intermediate model. For example... Figure 5 As shown, tasks 4, 5, and 6 are pre-selected as the core target tasks. The third input modal data in the third training sample includes three types of modal data: modal data 3, modal data 4, and modal data 5. The annotation information corresponding to the third input modal data includes the annotation information corresponding to the output results of task 4, task 5, and task 6. The third stage of model training ( Figure 5 (Middle gray area), the encoder and decoder in the intermediate model are trained based on at least one third training sample to obtain the target model.

[0115] The technical solution of this embodiment divides the model training phase into three stages. The first stage involves using a subset of task learning modules as the current target task learning module. Based on at least one first training sample, the first feature extraction network associated with that first training sample and the current target task learning module are trained in the original model to obtain a primary model. The second stage involves training the primary model as a whole based on at least one second training sample corresponding to all task learning modules. The third stage involves using at least one task learning module as a key task learning module. While training the intermediate model based on at least one third training sample, the training parameters of the first feature extraction network in the intermediate model are locked. This allows for adjustments to the training parameters of the second feature extraction network and the key task learning module in the intermediate model to obtain the target model. This approach allows for the use of different labeled data at different stages, reducing data labeling and training costs while accelerating model convergence.

[0116] Example 3

[0117] Figure 6 This is a flowchart of a first data processing method provided in an embodiment of the present invention. This embodiment is applicable to data processing situations. The method can be executed by a data processing device in this embodiment of the present invention, which can be implemented in software and / or hardware, such as... Figure 6 As shown, the method specifically includes the following steps:

[0118] S601. Obtain target multimodal data.

[0119] In this embodiment, the target multimodal data includes modal data of all modal types required for the target model to make predictions. Multimodal data refers to data from multiple sources, which may have different formats, structures, and representations. For example, the target multimodal data may include: driving-related data, which includes: sensor information and navigation planning information; the sensor information includes: environmental perception information and status information; the environmental perception information includes: current frame data and / or current point cloud data.

[0120] It should be noted that sensor information can be collected by sensors inside the vehicle or intelligent robot, and navigation planning information can be obtained by extracting navigation data based on the current location information of the vehicle.

[0121] S602. Input the target multimodal data into the target model to obtain the target output result set.

[0122] In this embodiment, the target model can be a multi-task end-to-end model. The target model can be trained using any of the model training methods provided in the above embodiments, and this embodiment does not impose any restrictions on it.

[0123] In this embodiment, the target output result set includes multiple target output results. For example, the target output result set may include 6 target output results, which are: first obstacle information, second obstacle information, target road structure, target obstacle trajectory, target decision information, and target driving trajectory.

[0124] Optionally, the target multimodal data includes: driving-related data of the current vehicle, or driving-related data of the intelligent robot. The driving-related data includes: sensor information and navigation planning information. The sensor information includes: environmental perception information and state information. The environmental perception information includes: current frame data and / or current point cloud data. The target output result set includes: first obstacle information, second obstacle information, target road structure, target obstacle trajectory, target decision information, and target driving trajectory.

[0125] In this embodiment, both the first obstacle information and the second obstacle information are current obstacle information. The target obstacle trajectory is the predicted trajectory of the obstacle over a future period of time; for example, the target obstacle trajectory could be the predicted trajectory of the obstacle over the next 8 seconds. The target driving trajectory is the predicted driving trajectory of a vehicle or intelligent robot over a future period of time; for example, the target driving trajectory could be the driving trajectory of a vehicle or intelligent robot over the next 8 seconds.

[0126] In this embodiment, the status information includes: the current vehicle's position information and / or the current vehicle's attitude information. The current vehicle's position information can be the position information collected by a GPS device deployed inside the current vehicle. The current vehicle's attitude information can include: the current vehicle's orientation, pitch angle, accelerator pedal opening, and gear information, etc. The navigation planning information can be the navigation planning information obtained by extracting navigation data (total navigation data from the starting point to the destination) based on the current vehicle's position information (navigation data from the current vehicle's position to the destination, or navigation data from a preset distance before the current vehicle's position to the destination).

[0127] In a specific example, end-to-end autonomous driving technology is a unified framework or method designed to integrate multiple tasks in autonomous driving. Its aim is to handle different driving tasks, such as perception, prediction, decision-making, and control, through a single model. Achieving multi-task output through an integrated model can improve overall system performance and collaborative efficiency, optimize the execution of autonomous driving tasks, and enhance the accuracy and reliability of each task, such as... Figure 7 As shown, the target model includes a backbone network, an encoder, and a decoder. The target multimodal data includes current frame data, current point cloud data, current vehicle position information, current vehicle attitude information, and navigation planning information. Inputting the current frame data, current point cloud data, current vehicle position information, current vehicle attitude information, and navigation planning information into the target model yields first obstacle information, second obstacle information, target road structure, target obstacle trajectory, target decision information, and the target vehicle's trajectory.

[0128] The technical solution of this embodiment obtains a target output result set by inputting the target multimodal data into the target model. Multi-task output is achieved through an integrated model. This reduces information loss and thus improves the accuracy of model prediction.

[0129] Example 4

[0130] Figure 8 This is a schematic diagram of a model training device provided in an embodiment of the present invention. This embodiment is applicable to model training scenarios. The device can be implemented using software and / or hardware, and can be integrated into any device that provides model training functionality, such as… Figure 8 As shown, the model training device specifically includes: a first training module 801, a second training module 802, and a third training module 803.

[0131] The first training module is used to train the first feature extraction network and the feature parsing network in the original model based on at least one first training sample to obtain a primary model. The first training sample includes first input modality data and the annotation information corresponding to the first input modality data.

[0132] The second training module is used to train the first feature extraction network, the second feature extraction network, and the feature parsing network in the primary model based on at least one second training sample to obtain an intermediate model. The second training sample includes second input modal data and the annotation information corresponding to the second input modal data.

[0133] The third training module is used to train the second feature extraction network and the feature parsing network in the intermediate model based on at least one third training sample to obtain the target model. The third training sample includes third input modality data and the annotation information corresponding to the third input modality data.

[0134] The above-mentioned products can execute the model training method provided in any embodiment of the present invention, and have the corresponding functional modules and beneficial effects of the execution method.

[0135] Example 5

[0136] Figure 9 This is a schematic diagram of a data processing device provided in an embodiment of the present invention. This embodiment is applicable to data processing applications. The device can be implemented using software and / or hardware, and can be integrated into any device that provides data processing functionality, such as… Figure 9 As shown, the data processing device specifically includes: a target multimodal data acquisition module 901 and a data processing module 902.

[0137] The target multimodal data acquisition module is used to acquire target multimodal data, wherein the target multimodal data includes modal data of all modal types required for the target model to make predictions.

[0138] The data processing module is used to input the target multimodal data into the target model to obtain the target output result set.

[0139] The above-mentioned products can execute the data processing methods provided in any embodiment of the present invention, and have the corresponding functional modules and beneficial effects of executing the methods.

[0140] Example 6

[0141] Figure 10 A schematic diagram of an electronic device 60 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0142] like Figure 10 As shown, the electronic device 60 includes at least one processor 61 and a memory, such as a read-only memory (ROM) 62 and a random access memory (RAM) 63, communicatively connected to the at least one processor 61. The memory stores computer programs executable by the at least one processor. The processor 61 can perform various appropriate actions and processes based on the computer program stored in the ROM 62 or loaded into the RAM 63 from storage unit 68. The RAM 63 may also store various programs and data required for the operation of the electronic device 60. The processor 61, ROM 62, and RAM 63 are interconnected via a bus 64. An input / output (I / O) interface 65 is also connected to the bus 64.

[0143] Multiple components in electronic device 60 are connected to I / O interface 65, including: input unit 66, such as keyboard, mouse, etc.; output unit 67, such as various types of monitors, speakers, etc.; storage unit 68, such as disk, optical disk, etc.; and communication unit 69, such as network card, modem, wireless transceiver, etc. Communication unit 69 allows electronic device 60 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0144] Processor 61 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 61 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 61 performs the various methods and processes described above, such as model training methods or data processing methods.

[0145] In some embodiments, the model training method or data processing method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 68. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 60 via ROM 62 and / or communication unit 69. When the computer program is loaded into RAM 63 and executed by processor 61, one or more steps of the model training method or data processing method described above may be performed. Alternatively, in other embodiments, processor 61 may be configured to perform the model training method or data processing method by any other suitable means (e.g., by means of firmware).

[0146] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0147] Computer programs used to implement the model training method or data processing method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The computer programs can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0148] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0149] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0150] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0151] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0152] In one embodiment, the present invention further includes a computer program product, which includes a computer program that, when executed by a processor, implements the model training method or data processing method of any embodiment of the present invention.

[0153] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0154] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0155] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A model training method, characterized in that, The model includes: The method comprises a first feature extraction network, a second feature extraction network, and a feature parsing network, wherein the method includes: The first feature extraction network and the feature parsing network in the original model are trained based on at least one first training sample to obtain a primary model. The first training sample includes first input modality data and the annotation information corresponding to the first input modality data. The first feature extraction network, the second feature extraction network, and the feature parsing network in the primary model are trained based on at least one second training sample to obtain an intermediate model. The second training sample includes second input modal data and the annotation information corresponding to the second input modal data. The second feature extraction network and the feature parsing network in the intermediate model are trained based on at least one third training sample to obtain the target model. The third training sample includes third input modality data and the annotation information corresponding to the third input modality data.

2. The method according to claim 1, characterized in that, The feature parsing network includes multiple task learning modules, which include a current target task learning module and a key task learning module. The union of the current target task learning module and the key task learning module constitutes the multiple task learning modules.

3. The method according to claim 2, characterized in that, The first feature extraction network and the feature parsing network in the original model are trained based on at least one first training sample to obtain a primary model, including: Based on at least one first training sample, the first feature extraction network associated with the at least one first training sample and the current target task learning module in the original model are trained to obtain a primary model; wherein, the first input modal data includes at least one input modal data corresponding to the current target task learning module, and the annotation information corresponding to the first input modal data includes at least one annotation information corresponding to the current target task learning module.

4. The method according to claim 2, characterized in that, The second feature extraction network and the feature parsing network in the intermediate model are trained based on at least one third training sample to obtain the target model, including: While training the intermediate model based on at least one third training sample, the training parameters of the first feature extraction network in the intermediate model are locked to adjust the training parameters of the second feature extraction network and the key task learning module in the intermediate model to obtain the target model. The third input modal data includes at least one input modal data corresponding to the key task learning module, and the annotation information corresponding to the third input modal data includes at least one annotation information corresponding to the key task learning module.

5. The method according to claim 1, characterized in that, The second input modal data contains all types of modal data required for the target model to make predictions. The first input modal data contains at least one type of modal data required for the target model to make predictions. The third input modal data contains at least one type of modal data required for the target model to make predictions. The modal data contained in the first input modal data and the modal data contained in the third input modal data are of different types. The union of the modal data contained in the first input modal data and the modal data contained in the third input modal data represents all types of modal data required for the target model to make predictions. The annotation information corresponding to the second input modal data contains all types of output results of the target model. The annotation information corresponding to the first input modal data contains at least one type of output result of the target model. The annotation information corresponding to the third input modal data contains at least one type of output result of the target model. The output results contained in the annotation information corresponding to the first input modal data and the output results contained in the annotation information corresponding to the third input modal data are of different types. The union of the output results contained in the annotation information corresponding to the first input modal data and the output results contained in the annotation information corresponding to the third input modal data represents all types of output results of the target model.

6. The method according to claim 1, characterized in that, The first input modal data includes: environmental perception samples, which include: frame samples and / or point cloud samples; the annotation information corresponding to the first input modal data includes: a first obstacle label, a second obstacle label, and a road structure label; the second input modal data includes: driving-related samples, which include: sensor samples and navigation planning samples; the sensor samples include: environmental perception samples and state samples; the annotation information corresponding to the second input modal data includes: a first obstacle label, a second obstacle label, a road structure label, an obstacle prediction trajectory, decision information, and a driving trajectory; the third input modal data includes: state samples and navigation planning samples; the state samples include: position samples and attitude samples; the annotation information corresponding to the third input modal data includes: obstacle prediction trajectory, decision information, and a driving trajectory.

7. The method according to claim 1, characterized in that, The first feature extraction network includes a backbone network, the second feature extraction network includes an encoder, and the feature parsing network includes a decoder.

8. A data processing method, characterized in that, include: Acquire target multimodal data, wherein the target multimodal data includes modal data of all modal types required for the target model to make predictions; The target multimodal data is input into the target model to obtain the target output result set.

9. The method according to claim 8, characterized in that, The target multimodal data includes: driving-related data of the current vehicle or driving-related data of the intelligent robot. The driving-related data includes: sensor information and navigation planning information. The sensor information includes: environmental perception information and state information. The environmental perception information includes: current frame data and / or current point cloud data. The target output result set includes: first obstacle information, second obstacle information, target road structure, target obstacle trajectory, target decision information, and target driving trajectory.

10. A model training device, characterized in that, The model includes: The model training device comprises a first feature extraction network, a second feature extraction network, and a feature parsing network, and includes: The first training module is used to train the first feature extraction network and the feature parsing network in the original model based on at least one first training sample to obtain a primary model. The first training sample includes first input modality data and the annotation information corresponding to the first input modality data. The second training module is used to train the first feature extraction network, the second feature extraction network, and the feature parsing network in the primary model based on at least one second training sample to obtain an intermediate model. The second training sample includes second input modal data and the annotation information corresponding to the second input modal data. The third training module is used to train the second feature extraction network and the feature parsing network in the intermediate model based on at least one third training sample to obtain the target model. The third training sample includes third input modality data and the annotation information corresponding to the third input modality data.

11. A data processing apparatus, characterized in that, include: The target multimodal data acquisition module is used to acquire target multimodal data, wherein the target multimodal data includes modal data of all modal types required for the target model to make predictions; The data processing module is used to input the target multimodal data into the target model to obtain the target output result set.

12. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that are used to cause a processor to execute the method of any one of claims 1-9.

14. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method according to any one of claims 1-9.

Images