Multi-task processing method based on large model, target large model and equipment

Abstract

The embodiment of the invention provides a multi-task processing method based on a large model, a target large model and equipment. The method comprises the following steps: inputting to-be-processed data into a target large model, and carrying out multi-task processing to obtain a task processing result of multiple tasks. Wherein the target large model comprises a plurality of backbone networks, and each backbone network is connected with a plurality of feature extraction units and task processing units. The method is used for achieving the effects of high multi-task processing speed and high efficiency. The method comprises the steps of inputting to-be-trained data into an initial large model, and performing multi-task processing training to obtain a target large model; wherein the initial large model comprises a plurality of backbone networks, each backbone network is connected with a plurality of feature extraction units to be trained and task processing units to be trained, and parameters of the backbone networks are not updated in the training process. The method is used for achieving the effects of high training speed and high efficiency of the large model for multi-task processing.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a large model-based multi-task processing method, a target large model and a device. Background Art

[0002] Multi-task models can be applied to multi-task processing of data. For example, for images, when multiple tasks such as person recognition, object classification, and target segmentation need to be processed, the multi-task model can be used to complete the task processing of the image according to the requirements of the multiple task processing;

[0003] However, in the prior art, when the task processing is completed based on the multi-task model, the branches in the multi-task model will affect each other, thereby slowing down the task processing speed and causing low efficiency.

[0004] Therefore, there is an urgent need for a solution that can quickly process data to quickly obtain task processing results. Summary of the invention

[0005] The embodiments of the present application provide a large model-based multi-task processing method, a target large model and a device to achieve the effect of fast multi-task processing speed and high efficiency.

[0006] In a first aspect, an embodiment of the present application provides a multi-task processing method based on a large model, comprising:

[0007] The data to be processed and the target processing task are input into the target large model, and the following processing is performed through the target large model:

[0008] Extracting features of the data to be processed by the target backbone network in the target large model to obtain a first feature of the data to be processed; associating the target backbone network with the data modality of the data to be processed;

[0009] The first feature is extracted by a feature extraction unit in a target task network to obtain a second feature of the data to be processed; the target task network is connected to the target backbone network, and the target task network is a task network associated with the target processing task, and different task networks connected to the backbone network are associated with different processing tasks;

[0010] And the target processing task is executed on the second feature by the task processing unit in the target task network to obtain the task processing result of the data to be processed.

[0011] In one possible implementation, the target large model includes M backbone networks, and different backbone networks are associated with different data modalities; wherein the backbone network is connected to N task networks, and different task networks connected to the backbone network are associated with different processing tasks; the task network includes a feature extraction unit and a task processing unit connected in sequence; M is an integer greater than 0; and N is an integer greater than 0.

[0012] In a possible implementation, the method further includes:

[0013] If it is determined that the target task network is not included in the available task networks connected to the target backbone network, prompt information for the target processing task is generated and issued; wherein the prompt information indicates that the target processing task cannot be processed.

[0014] In a possible implementation, the data modality includes at least one of image, video, audio, document, text, and point cloud data.

[0015] In a possible implementation, when the data modality includes an image or a video, the processing task includes at least one of target recognition, target classification, event recognition, behavior recognition, posture recognition, emotion recognition, target segmentation, key point detection, and data format conversion;

[0016] When the data modality includes audio, the processing task includes at least one of speech recognition, voice dialogue, voice inquiry, data format conversion, speech emotion recognition, speech-to-text conversion, and speech summary generation;

[0017] When the data modality includes documents, the processing task includes at least one of document classification, key information extraction, document summary generation, document sentiment analysis, and document security detection;

[0018] When the data modality includes text, the processing task includes at least one of text classification, entity recognition, generating text summaries, text translation, text deduplication, text sentiment analysis, and format error recognition;

[0019] When the data modality includes point cloud data, the processing task includes at least one of target recognition, target segmentation, key point detection, point cloud denoising, and point cloud completion.

[0020] In a second aspect, an embodiment of the present application provides a target large model for multi-task processing, including:

[0021] M backbone networks, different backbone networks are associated with different data modalities; the backbone networks are used to extract a first feature from the to-be-processed data of the associated data modality;

[0022] The backbone network is connected to N task networks, and different task networks connected to the backbone network are associated with different processing tasks; the task network includes a feature extraction unit and a task processing unit connected in sequence; the feature extraction unit is used to extract a first feature extracted by the connected backbone network to obtain a second feature; the task processing unit is used to perform an associated processing task on the second feature extracted by the connected feature extraction unit to obtain a task processing result of the data to be processed;

[0023] Wherein, the M is an integer greater than 0, and the N is an integer greater than 0.

[0024] In a possible implementation, the feature extraction unit includes a multi-scale feature extraction module and a first fusion module;

[0025] The multi-scale feature extraction module is used to extract the first feature extracted by the connected backbone network to obtain data features in multiple dimensions;

[0026] The first fusion module is used to perform feature fusion on the data features in the multiple dimensions to obtain the second features.

[0027] In a possible implementation, the feature extraction unit includes an upsampling module, a downsampling module, and a second fusion module;

[0028] The upsampling module is used to perform high-level feature extraction on the first feature extracted by the connected backbone network to obtain high-level semantic feature information;

[0029] The downsampling module is used to perform low-level feature extraction on the first feature extracted by the connected backbone network to obtain low-level semantic feature information;

[0030] The second fusion module is used to perform feature fusion on the high-level semantic feature information and the low-level semantic feature information to obtain a second feature.

[0031] In a possible implementation, the data modality includes at least one of image, video, audio, document, text, and point cloud data.

[0032] In a possible implementation, when the data modality includes an image or a video, the processing task includes at least one of target recognition, target classification, event recognition, behavior recognition, posture recognition, emotion recognition, target segmentation, key point detection, and data format conversion;

[0033] When the data modality includes audio, the processing task includes at least one of speech recognition, voice dialogue, voice inquiry, data format conversion, speech emotion recognition, speech-to-text conversion, and speech summary generation;

[0034] When the data modality includes documents, the processing task includes at least one of document classification, key information extraction, document summary generation, document sentiment analysis, and document security detection;

[0035] When the data modality includes text, the processing task includes at least one of text classification, entity recognition, generating text summaries, text translation, text deduplication, text sentiment analysis, and format error recognition;

[0036] When the data modality includes point cloud data, the processing task includes at least one of target recognition, target segmentation, key point detection, point cloud denoising, and point cloud completion.

[0037] In a third aspect, an embodiment of the present application provides a training method for a large model for multi-task processing, including:

[0038] The data to be trained and the training processing tasks are input into the initial large model, and the following processing is performed by the initial large model:

[0039] Extracting features of the data to be trained by the target backbone network in the initial large model to obtain a first feature of the data to be trained; associating the target backbone network with the data modality of the data to be trained;

[0040] Based on the first feature, the task processing result of the data to be trained is determined by the feature extraction unit and the task processing unit connected in sequence in the target task network, and the parameters of the feature extraction unit and the task processing unit in the target task network are adjusted based on the task processing result; the target task network is connected to the target backbone network, and the target task network is the task network associated with the training processing task, and different task networks connected to the backbone network are associated with different processing tasks.

[0041] In one possible implementation, the initial large model includes M backbone networks, and different backbone networks are associated with different data modalities; wherein the backbone network is connected to N task networks, and different task networks connected to the backbone network are associated with different processing tasks; the task network includes a feature extraction unit and a task processing unit connected in sequence; M is an integer greater than 0; and N is an integer greater than 0.

[0042] In a possible implementation, based on the first feature, determining the task processing result of the to-be-trained data by sequentially connecting a feature extraction unit and a task processing unit in a target task network includes:

[0043] Extracting the first feature by a feature extraction unit in the target task network to obtain a second feature of the data to be trained;

[0044] Based on the task processing unit connected to the feature extraction unit in the target task network, the training processing task is performed on the second feature to obtain the task processing result of the data to be trained.

[0045] In a possible implementation, the to-be-trained data has a calibrated task processing result; and adjusting parameters of a feature extraction unit and a task processing unit in the target task network based on the task processing result includes:

[0046] Obtaining a loss value of a loss function according to the calibrated task processing result of the to-be-trained data and the task processing result output by the target task network;

[0047] According to the loss value of the loss function, the parameters of the feature extraction unit and the task processing unit in the target task network are updated.

[0048] In a possible implementation, the data modality includes at least one of image, video, audio, document, text, and point cloud data.

[0049] In a possible implementation, when the data modality includes an image or a video, the processing task includes at least one of target recognition, target classification, event recognition, behavior recognition, posture recognition, emotion recognition, target segmentation, key point detection, and data format conversion;

[0050] When the data modality includes audio, the processing task includes at least one of speech recognition, voice dialogue, voice inquiry, data format conversion, speech emotion recognition, speech-to-text conversion, and speech summary generation;

[0051] When the data modality includes documents, the processing task includes at least one of document classification, key information extraction, document summary generation, document sentiment analysis, and document security detection;

[0052] When the data modality includes text, the processing task includes at least one of text classification, entity recognition, generating text summaries, text translation, text deduplication, text sentiment analysis, and format error recognition;

[0053] When the data modality includes point cloud data, the processing task includes at least one of target recognition, target segmentation, key point detection, point cloud denoising, and point cloud completion.

[0054] In a fourth aspect, an embodiment of the present application provides a multi-task processing device based on a large model, comprising:

[0055] An input unit, used to input the data to be processed and the target processing task into the target large model, and execute the processing process of the following units through the target large model;

[0056] A first extraction unit is used to extract features of the data to be processed through a target backbone network in the target large model to obtain a first feature of the data to be processed; the target backbone network is associated with a data modality of the data to be processed;

[0057] a second extraction unit, configured to extract the first feature through a feature extraction unit in a target task network to obtain a second feature of the data to be processed; the target task network is connected to the target backbone network, and the target task network is a task network associated with the target processing task, and different task networks connected to the backbone network are associated with different processing tasks;

[0058] An execution unit is used to execute the target processing task on the second feature through a task processing unit in the target task network to obtain a task processing result of the data to be processed.

[0059] In one possible implementation, the target large model includes M backbone networks, and different backbone networks are associated with different data modalities; wherein the backbone network is connected to N task networks, and different task networks connected to the backbone network are associated with different processing tasks; the task network includes a feature extraction unit and a task processing unit connected in sequence; M is an integer greater than 0; and N is an integer greater than 0.

[0060] In a possible implementation, the large model-based multi-task processing device 70 further includes:

[0061] The prompt unit is used to generate and issue prompt information for the target processing task if it is determined that the available task networks connected to the target backbone network do not include the target task network; wherein the prompt information indicates that the target processing task cannot be processed.

[0062] In a possible implementation, the data modality includes at least one of image, video, audio, document, text, and point cloud data.

[0063] In a possible implementation, when the data modality includes images or videos, the processing task includes at least one of target recognition, target classification, event recognition, behavior recognition, gesture recognition, emotion recognition, target segmentation, and data format conversion;

[0064] When the data modality includes audio, the processing task includes at least one of speech recognition, voice dialogue, voice inquiry, data format conversion, speech emotion recognition, speech-to-text conversion, and speech summary generation;

[0065] When the data modality includes documents, the processing task includes at least one of document classification, key information extraction, document summary generation, document sentiment analysis, and document security detection;

[0066] When the data modality includes text, the processing task includes at least one of text classification, entity recognition, generating text summaries, text translation, text deduplication, text sentiment analysis, and format error recognition;

[0067] When the data modality includes point cloud data, the processing task includes at least one of target recognition, target segmentation, point cloud denoising, and point cloud completion.

[0068] In a fifth aspect, an embodiment of the present application provides a training device for a large model for multi-task processing, comprising:

[0069] An input unit, used to input the data to be trained and the training processing task into the initial large model, and execute the processing process of the following units through the initial large model;

[0070] An extraction unit, configured to extract features of the data to be trained by using a target backbone network in the initial large model to obtain a first feature of the data to be trained; the target backbone network is associated with a data modality of the data to be trained;

[0071] a determination unit, configured to determine, based on the first feature, a task processing result of the data to be trained through a feature extraction unit and a task processing unit sequentially connected in a target task network; the target task network is connected to the target backbone network, and the target task network is a task network associated with the training processing task, and different task networks connected to the backbone network are associated with different processing tasks;

[0072] An adjustment unit is used to adjust parameters of a feature extraction unit and a task processing unit in the target task network based on the task processing result.

[0073] In a sixth aspect, an embodiment of the present application provides another large model training device for multi-task processing, comprising:

[0074] An input unit, used to input the data to be trained and the training processing task into the initial large model, and execute the processing process of the following units through the initial large model;

[0075] An extraction unit, configured to extract features of the data to be trained by using a target backbone network in the initial large model to obtain a first feature of the data to be trained; the target backbone network is associated with a data modality of the data to be trained;

[0076] a determination unit, configured to determine, based on the first feature, a task processing result of the data to be trained through a feature extraction unit and a task processing unit sequentially connected in a target task network; the target task network is connected to the target backbone network, and the target task network is a task network associated with the training processing task, and different task networks connected to the backbone network are associated with different processing tasks;

[0077] An adjustment unit is used to adjust parameters of a feature extraction unit and a task processing unit in the target task network based on the task processing result.

[0078] In one possible implementation, the initial large model includes M backbone networks, and different backbone networks are associated with different data modalities; wherein the backbone network is connected to N task networks, and different task networks connected to the backbone network are associated with different processing tasks; the task network includes a feature extraction unit and a task processing unit connected in sequence; M is an integer greater than 0; and N is an integer greater than 0.

[0079] In a possible implementation manner, the determining unit is specifically configured to:

[0080] An extraction module, used for extracting the first feature through a feature extraction unit in a target task network to obtain a second feature of the data to be trained;

[0081] An execution module is used to execute the training processing task on the second feature based on a task processing unit connected to the feature extraction unit in the target task network to obtain a task processing result of the data to be trained.

[0082] In a possible implementation, the data to be trained has calibrated task processing results;

[0083] In a possible implementation manner, the adjustment unit is specifically configured to:

[0084] A determination module, used to obtain a loss value of a loss function according to a calibrated task processing result of the to-be-trained data and a task processing result output by the target task network;

[0085] An updating module is used to update the parameters of the feature extraction unit and the task processing unit in the target task network according to the loss value of the loss function.

[0086] In a possible implementation, the data modality includes at least one of image, video, audio, document, text, and point cloud data.

[0087] In a possible implementation, when the data modality includes images or videos, the processing task includes at least one of target recognition, target classification, event recognition, behavior recognition, gesture recognition, emotion recognition, target segmentation, and data format conversion;

[0088] When the data modality includes audio, the processing task includes at least one of speech recognition, voice dialogue, voice inquiry, data format conversion, speech emotion recognition, speech-to-text conversion, and speech summary generation;

[0089] When the data modality includes documents, the processing task includes at least one of document classification, key information extraction, document summary generation, document sentiment analysis, and document security detection;

[0090] When the data modality includes text, the processing task includes at least one of text classification, entity recognition, generating text summaries, text translation, text deduplication, text sentiment analysis, and format error recognition;

[0091] When the data modality includes point cloud data, the processing task includes at least one of target recognition, target segmentation, point cloud denoising, and point cloud completion.

[0092] In a seventh aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;

[0093] The memory stores computer-executable instructions;

[0094] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the first aspect as above, and / or various possibilities of the first aspect; or, the processor executes the second aspect as above, and / or various possibilities of the second aspect for processing the target large model; or, the processor executes the third aspect as above, and / or various possibilities of the third aspect.

[0095] In an eighth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer execution instructions, and when the computer execution instructions are executed by a processor, they are used to implement the first aspect as above, and / or various possibilities of the first aspect; or, when the computer execution instructions are executed by a processor, they are used to implement the second aspect as above, and / or various possible target large model processing processes of the second aspect; or, when the computer execution instructions are executed by a processor, they are used to implement the third aspect as above, and / or various possibilities of the third aspect.

[0096] In the ninth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the first aspect as above, and / or various possibilities of the first aspect; or, when executed by a processor, implements the second aspect as above, and / or various possible target large model processing processes of the second aspect; or, when executed by a processor, implements the third aspect as above, and / or various possibilities of the third aspect.

[0097] The embodiments of the present application provide a large model-based multi-task processing method, a target large model, and a device. By inputting the data to be processed and the target processing task into the target large model, the target large model performs the following processing: extracting features of the data to be processed through the target backbone network in the target large model to obtain a first feature of the data to be processed; associating the target backbone network with the data modality of the data to be processed; then, extracting features of the first feature through the feature extraction unit in the target task network to obtain a second feature of the data to be processed; connecting the target task network to the target backbone network, and the target task network is a task network associated with the target processing task, and different task networks connected to the backbone network are associated with different processing tasks; and executing the target processing task on the second feature through the task processing unit in the target task network to obtain a task processing result of the data to be processed.

[0098] Since the target large model includes multiple feature extraction units and task processing units corresponding to each feature extraction unit. Based on the feature extraction units and task processing units corresponding to each task, each task processing result is obtained. Moreover, in the process of multi-task processing based on the target large model, when different backbone networks extract the first feature of data of different data modes in the processed data, the backbone networks do not affect each other; when the feature extraction units in the target task network associated with each target processing task extract the second feature from the first feature, the feature extraction units do not affect each other. When the task processing units in the target task network associated with each target processing task execute the target processing task on the second feature to obtain the task processing result, the task processing units do not affect each other. Therefore, this embodiment can provide a multi-task parallel processing method based on a large model. Furthermore, each task is processed quickly and efficiently.

[0099] Since a plurality of backbone networks may be included in the target large model, and the target backbone network is associated with the data modality of the data to be processed, this embodiment can realize multi-task processing of the data to be processed of multiple data modalities at the same time.

[0100] Since in the target large model, each target task network associated with each target processing task has a feature extraction unit for extracting the second feature required by the task processing unit in the target task network, the semantic feature information required by the task processing unit can be extracted more accurately, thereby obtaining a higher accuracy rate of the task processing result.

[0101] Since the target processing task can be one or more tasks in the target large model, and each task is independent of each other, it is possible to execute only part of the tasks in the target large model for the data to be processed, thereby improving the flexibility of model application. BRIEF DESCRIPTION OF THE DRAWINGS

[0102] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0103] Figure 1 An architecture diagram of a target large model for multi-tasking provided by this application;

[0104] Figure 2 Another architecture diagram of a target large model for multi-tasking provided by the present application;

[0105] Figure 3 A flowchart of a training method for a large model for multi-task processing provided in this application;

[0106] Figure 4 A flowchart of a multi-task processing method based on a large model provided in this application;

[0107] Figure 5 A schematic diagram of the structure of a multi-task processing device based on a large model provided in this application;

[0108] Figure 6 A schematic diagram of the structure of a large model training device for multi-task processing provided by the present application;

[0109] Figure 7 A schematic diagram of the structure of another large model training device for multi-task processing provided by the present application;

[0110] Figure 8 A schematic diagram of the structure of the electronic device provided in this application.

[0111] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0112] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0113] With the continuous development of artificial intelligence technology, multi-task models can be applied to multi-task processing of data. For example, during autonomous driving, multiple tasks can be processed on the images obtained by the vehicle based on the multi-task model according to the requirements of autonomous driving, such as identifying obstacle categories and obstacle detection frames.

[0114] It is understandable that whenever multiple tasks need to be processed on data, multiple tasks can be processed on data based on the multi-task model according to the requirements of the multiple tasks. This embodiment does not limit the application scenarios of the multi-task model, the specific data modality, and the specific processing tasks.

[0115] In one example, when multiple tasks are processed based on a multi-task model, the coupling degree of different tasks in the multi-task model is high, resulting in slow task processing and low efficiency.

[0116] The present application provides a large model-based multi-task processing method, a target large model and a device to solve the above problems.

[0117] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0118] An embodiment of the present application provides a target large model for multi-task processing, which includes: M backbone networks, different backbone networks are associated with different data modalities; the backbone network is used to extract a first feature from the data to be processed of the associated data modality; the backbone network is connected to N task networks, and different task networks connected to the backbone network are associated with different processing tasks; the task network includes a feature extraction unit and a task processing unit connected in sequence; the feature extraction unit is used to extract a first feature extracted by the connected backbone network to obtain a second feature; the task processing unit is used to perform an associated processing task on the second feature extracted by the connected feature extraction unit to obtain a task processing result of the data to be processed; wherein M is an integer greater than 0, and N is an integer greater than 0.

[0119] Exemplarily, the target large model includes: M backbone networks, each backbone network is connected to N task networks, and each task network includes a feature extraction unit and a task processing unit connected in sequence; wherein different backbone networks are associated with different data modalities, and different task networks connected to each backbone network are associated with different processing tasks; the backbone network is used to extract the first feature of the data to be processed of the associated data modality, and then the feature extraction unit in the task network connected to the backbone network is used to extract the first feature extracted by the connected backbone network to obtain the second feature, and the task processing unit in the task network connected to the backbone network is used to perform the associated processing task on the second feature extracted by the connected feature extraction unit to obtain the task processing result of the data to be processed.

[0120] The data to be processed may be, for example, one or any combination of video data, point cloud data, audio data, etc. The data to be processed may be obtained through a sensor or a built-in microphone of a device, and this embodiment does not limit the method of obtaining the data to be processed.

[0121] In a feasible implementation, the data modality includes at least one of image, video, audio, document, text, and point cloud data.

[0122] Exemplarily, the data modality in the embodiments of the present application refers to the type or form of data. Different data modalities have different characteristics and uses. Technicians in this field may set one or more data modalities according to actual needs. For example, the data modality may include but is not limited to one or any combination of images, videos, audio, documents, text, point cloud data, etc.

[0123] After receiving the data to be processed, the target large model can determine the data modality according to the data format contained in the data to be processed. For example, according to the data format of the video data, the data modality of the video data in the data to be processed is determined to be video; according to the data format of the point cloud data, the data modality of the point cloud data in the data to be processed is determined to be point cloud data; according to the data format of the audio data, the data modality of the audio data in the data to be processed is determined to be audio.

[0124] Afterwards, in the target large model, the backbone network associated with the video data modality is used to extract the first feature from the video data in the data to be processed, and then the feature extraction unit in the task network connected to the backbone network is used to extract the first feature extracted by the connected backbone network to obtain the second feature, and the task processing unit in the task network connected to the backbone network is used to perform the associated processing task on the second feature extracted by the connected feature extraction unit to obtain the task processing result of the video data in the data to be processed. The target large model processes the point cloud data modality and the audio data modality in a similar way to the target large model processes the video data modality, which will not be repeated here.

[0125] In one example, the first feature may be a high-level abstraction of the data to be processed. These feature information do not directly correspond to the original form of the data to be processed, but represent key attributes or patterns in the data to be processed. These feature information are not designed for a specific processing task, but have sufficient generalization capabilities and can be used in multiple different processing tasks.

[0126] In one example, the second feature may be semantic feature information at a higher level of abstraction used to perform task processing on a task processing unit in a task network to process data. For example, in an obstacle detection task for video data, the second feature may include the category of the obstacle, such as a vehicle or a pedestrian; it may also include the shape of the obstacle, such as an outer contour box of a vehicle or a pedestrian; it may also include the position of the obstacle, such as the position of the outer contour box of a vehicle in the image or the position of the outer contour box of a pedestrian in the image.

[0127] In a feasible implementation, when the data modality includes images or videos, the processing tasks include at least one of target recognition, target classification, event recognition, behavior recognition, posture recognition, emotion recognition, target segmentation, key point detection, and data format conversion;

[0128] When the data modality includes audio, the processing tasks include at least one of speech recognition, voice dialogue, voice inquiry, data format conversion, speech emotion recognition, speech-to-text conversion, and speech summary generation;

[0129] When the data modality includes documents, the processing tasks include at least one of document classification, key information extraction, document summary generation, document sentiment analysis, and document security detection;

[0130] When the data modality includes text, the processing tasks include at least one of text classification, entity recognition, text summary generation, text translation, text deduplication, text sentiment analysis, and format error recognition;

[0131] When the data modality includes point cloud data, the processing tasks include at least one of target recognition, target segmentation, key point detection, point cloud denoising, and point cloud completion.

[0132] Exemplarily, a processing task may refer to a task performed on the data to be processed based on a target macro model. A person skilled in the art may set one or more processing tasks according to actual needs. Furthermore, the processing tasks associated with different data modalities may be the same, different, or partially the same. A person skilled in the art may set them according to actual needs. For example:

[0133] When the data modality includes images or videos, the processing tasks may include, but are not limited to, at least one of target recognition, target classification, event recognition, behavior recognition, gesture recognition, emotion recognition, target segmentation, key point detection, and data format conversion;

[0134] When the data modality includes audio, the processing task may include, but is not limited to, at least one of speech recognition, voice dialogue, voice inquiry, data format conversion, speech emotion recognition, speech-to-text conversion, and speech summary generation;

[0135] When the data modality includes documents, the processing tasks may include but are not limited to at least one of document classification, key information extraction, document summary generation, document sentiment analysis, and document security detection;

[0136] When the data modality includes text, the processing task may include but is not limited to at least one of text classification, entity recognition, generating text summaries, text translation, text deduplication, text sentiment analysis, and format error recognition;

[0137] When the data modality includes point cloud data, the processing tasks include at least one of target recognition, target segmentation, key point detection, point cloud denoising, and point cloud completion.

[0138] In a feasible implementation, the feature extraction unit in the target large model includes a multi-scale feature extraction module and a first fusion module;

[0139] The multi-scale feature extraction module is used to extract the first feature extracted by the connected backbone network to obtain data features in multiple dimensions;

[0140] The first fusion module is used to fuse the data features in multiple dimensions to obtain the second features.

[0141] Exemplarily, each feature extraction unit may include a multi-scale feature extraction module and a first fusion module connected in sequence. The multi-scale feature extraction module is used to extract features from the first feature extracted by the connected backbone network to obtain data features in multiple dimensions. Among them, the data features in multiple dimensions are, for example, feature information extracted by the target large model at different abstract levels. In larger dimensions, the feature information contains more contextual information, which helps the model understand the global structure of the scene; in smaller dimensions, the feature information contains more detailed information, which helps the target large model recognize the specific attributes and characteristics of the object.

[0142] The first fusion module is used to perform feature fusion on the multi-dimensional data features extracted by the connected multi-scale feature extraction module based on the content required by the connected task processing unit to process the task, so as to obtain the second feature. Feature fusion can be understood as fusing features of different dimensions, so as to obtain richer feature information for the connected task processing unit to process the task.

[0143] In a feasible implementation, the feature extraction unit in the target large model includes an upsampling module, a downsampling module, and a second fusion module;

[0144] The upsampling module is used to perform high-level feature extraction on the first feature extracted by the connected backbone network to obtain high-level semantic feature information;

[0145] The downsampling module is used to perform low-level feature extraction on the first feature extracted by the connected backbone network to obtain low-level semantic feature information;

[0146] The second fusion module is used to perform feature fusion on the high-level semantic feature information and the low-level semantic feature information to obtain the second feature.

[0147] Exemplarily, each feature extraction unit may include an upsampling module, a downsampling module, and a second fusion module connected in sequence. The upsampling module is used to perform high-level feature extraction on the first feature extracted by the connected backbone network to obtain high-level semantic feature information. For example, for video data, the high-level semantic feature information may be a category (e.g., a vehicle, a pedestrian), a position (e.g., the position of a vehicle in a video image, the position of a pedestrian in a video image), etc. The downsampling module is used to perform low-level feature extraction on the first feature extracted by the connected backbone network to obtain low-level semantic feature information. The low-level semantic feature information may be, for example, a relatively basic feature directly extracted from the data to be processed. For example, for video data, the low-level semantic feature information may be a color feature, a texture feature, an edge feature, etc.

[0148] The second fusion module is used to perform feature fusion on the high-level semantic feature information and the low-level semantic feature information based on the content required by the connected task processing unit to process the task, and obtain the second feature. Feature fusion can be understood as fusing the features of the high-level semantic feature information and the low-level semantic feature, so as to obtain the second feature with richer feature information, which is used for the connected task processing unit to process the task.

[0149] As an example, Figure 1 The following is an architecture diagram of a target large model for multi-tasking provided by this application. Figure 1 The illustrated target large model 10 includes a backbone network 101, and the backbone network 101 is connected to a number of task networks (such as but not limited to the task network 102-1, task network 102-2, task network 102-3, etc. illustrated in the figure). Task network 102-1 includes a feature extraction unit 102-1-1 and a task processing unit 102-1-2 connected in sequence. Task network 102-2 includes a feature extraction unit 102-2-1 and a task processing unit 102-2-2 connected in sequence. Task network 102-3 includes a feature extraction unit 102-3-1 and a task processing unit 102-3-2 connected in sequence.

[0150] Among them, the backbone network 101 is associated with a data modality, and the backbone network 101 is used to extract the first feature of the data to be processed of the associated data modality. The task network 102-1, the task network 102-2 and the task network 102-3 are associated with different processing tasks. For the task network 102-1, the feature extraction unit 102-1-1 in the task network 102-1 is used to extract the first feature extracted by the connected backbone network 101 to obtain the second feature, and the task processing unit 102-1-2 in the task network 102-1 is used to perform the associated processing task on the second feature extracted by the connected feature extraction unit 102-1-1 to obtain the task processing result of the data to be processed. The functions of the task network 102-2 and the task network 102-3 are similar to those of the task network 102-1, and will not be repeated here.

[0151] In a feasible implementation, in the target large model, one or more task networks connected to the backbone network can be added (or opened) (for example, in Figure 1 The target model 10 adds a task network), and the added task network connected to the backbone network ( Figure 1 (not shown) is used to execute the processing tasks associated with the task network on the to-be-processed data of the data modality associated with the backbone network.

[0152] And / or, in the target large model, one or more task networks connected to the backbone network can be modified (e.g. Figure 1In the target large model, the task network 102-1 is modified, and the modified task network connected to the backbone network is used to perform the processing tasks associated with the modified task network on the to-be-processed data of the data modality associated with the backbone network. The modified task network includes a feature extraction unit in the modified task network, and / or a task processing unit in the modified task network.

[0153] And / or, in the target large model, one or more task networks connected to the backbone network can be deleted (or closed) (for example, in Figure 1 In the target large model, the task network 102-3 is deleted, and the deleted (or closed) task network connected to the backbone network is used to shield the to-be-processed data of the data modality associated with the backbone network from executing the processing tasks associated with the task network.

[0154] Furthermore, in the target macro model, when adding (or opening), and / or modifying, and / or deleting (or closing) a task network connected to the backbone network, the functions of other task networks connected to the backbone network are not affected.

[0155] As an example, Figure 2 Another architecture diagram of a target large model for multi-tasking provided by the present application; Figure 2 The target large model 20 shown in the figure includes at least two backbone networks (such as but not limited to Figure 2 (a) shows the backbone network 201, Figure 2 (b) shows the backbone network 203, etc.) Figure 2 (a) shows that the backbone network 201 connects several task networks (such as but not limited to Figure 2 (a) shows the task network 202-1. Figure 2 (a) shows the task network 202-2. Figure 2 (a) shows the task network 202-3, etc.) Figure 2 The task network 202-1 shown in (a) includes Figure 2 (a) shows the feature extraction unit 202-1-1 and Figure 2 (a) shows the task processing unit 202-1-2. Figure 2 The task network 202-2 shown in (a) includes Figure 2 (a) shows the feature extraction unit 202-2-1 and Figure 2 (a) shows the task processing unit 202-2-2. Figure 2 The task network 202-3 shown in (a) includes Figure 2 (a) shows the feature extraction unit 202-3-1 and Figure 2 (a) shows the task processing unit 202-3-2.

[0156] Figure 2 (b) shows that the backbone network 203 connects several task networks (such as but not limited to Figure 2 (b) shows the task network 204-1. Figure 2 (b) shows the task network 204-2. Figure 2 (b) illustrates the task network 204-3, etc.). Figure 2 (b) shows that the task network 204-1 includes Figure 2 (b) shows the feature extraction unit 204-1-1 and Figure 2 (b) shows the task processing unit 204-1-2. Figure 2 (b) shows that the task network 204-2 includes Figure 2 (b) shows the feature extraction unit 204-2-1 and Figure 2 (b) shows the task processing unit 204-2-2. Figure 2 (b) shows that the task network 204-3 includes Figure 2 (b) shows the feature extraction unit 204-3-1 and Figure 2 (b) shows the task processing unit 204-3-2.

[0157] in, Figure 2 (a) shows that the backbone network 201 is associated with a data mode. Figure 2 (a) shows that the backbone network 201 is used to extract the first feature of the to-be-processed data of the associated data modality. Figure 2 (a) shows the task network 202-1. Figure 2 (a) shows the task network 202-2 and Figure 2 (a) in the figure shows that the task network 202-3 in the figure associates different processing tasks. Figure 2 (a) shows the task network 202-1. Figure 2 (a) shows that the feature extraction unit 202-1-1 in the task network 202-1 is used to connect Figure 2 (a) shows that the first feature extracted by the backbone network 201 is extracted to obtain the second feature. Figure 2 (a) shows that the task processing unit 202-1-2 in the task network 202-1 is used to connect Figure 2(a) shows that the second feature extracted by the feature extraction unit 202-1-1 executes the associated processing task to obtain the task processing result of the data to be processed. Figure 2 (a) shows the task network 202-2 and Figure 2 (a) shows the function and Figure 2 The functions of the task network 202 - 1 shown in (a) are similar and will not be described again here.

[0158] Figure 2 (b) shows that the backbone network 203 is associated with another data mode, Figure 2 (b) shows the backbone network 203. Figure 2 (b) shows the task network 204-1. Figure 2 (b) shows the task network 204-2 and Figure 2 (b) illustrates the functions and Figure 2 (a) shows the backbone network 201, Figure 2 (a) shows the task network 202-1. Figure 2 The functions of the task network 202 - 2 shown in (a) are similar and will not be described again here.

[0159] In a feasible implementation, in the target large model, one or more backbone networks and their connected task networks can be added (or enabled) (for example, Figure 1 Add the content to Figure 2 In the target large model 20), the added (or opened) backbone network and the task network connected thereto are used to perform multi-task processing on the to-be-processed data of the data modality associated with the backbone network.

[0160] And / or, in the target large model, one or more backbone networks may be modified (e.g., modifying Figure 2 (b) shows the backbone network 203), and the modified backbone network is used to extract the first feature of the to-be-processed data of the data modality associated with the modified backbone network.

[0161] And / or, in the target large model, one or more backbone networks and their connected task networks can be deleted (or closed) (e.g. Figure 2 In the target large model, delete Figure 2 (b)), the deleted (or closed) backbone network and its connected task network are used to shield the multi-tasking processing of the to-be-processed data of the data modality associated with the backbone network.

[0162] And / or, in the target large model, one or more task networks can be added (or enabled) (e.g. Figure 2 In the target large model, Figure 2 (a) shows that a task network is added. The added (or enabled) task network is used to perform the processing task associated with the task network on the to-be-processed data of the data modality associated with the backbone network to which it is connected. The added (or enabled) task network can be connected to the same backbone network or to different backbone networks.

[0163] And / or, in the target large model, one or more task networks can be modified (e.g. Figure 2 In the target large model, modify Figure 2 (a) shows the task network 202-1), the modified task network is used to perform the processing task associated with the modified task network on the to-be-processed data of the data modality associated with the backbone network to which it is connected. The modified task network can be connected to the same backbone network or to different backbone networks.

[0164] And / or, in the target large model, one or more task networks can be deleted (or closed) (for example, in Figure 2 In the target large model, Figure 2 In (a), a task network 202-3 is deleted, and the deleted (or closed) task network is used to shield the processing tasks associated with the task network from being executed for the data to be processed of the data modality associated with the backbone network to which it is connected. The deleted task network can be connected to the same backbone network or to different backbone networks.

[0165] Moreover, in the target large model, when adding (or opening), and / or deleting (or closing) a backbone network and its connected task networks, and / or modifying a backbone network, the functions of other backbone networks and their connected task networks are not affected; when adding (or opening), and / or modifying, and / or deleting (or closing) a task network, the functions of other task networks are not affected.

[0166] As an example, it is possible but not limited to Figure 1 or Figure 2 The illustrated target large model is deployed to the corresponding execution subject to implement the multi-task processing method through the target large model; the execution subject of this embodiment can be a server, a distributed system, a terminal device, other electronic equipment / computer equipment, or any other device or equipment that can implement the solution of this application, without limitation. The server can be an independent server, or a server cluster, such as but not limited to any form including cloud servers, distributed servers, blockchain servers, Internet of Things servers, and intelligent driving platforms connected to vehicle-connected systems.

[0167] Figure 3 A flowchart of a training method for a large model for multi-task processing provided in this application is shown in FIG. Figure 3 As shown, the method includes:

[0168] S301, input the data to be trained and the training processing tasks into the initial large model.

[0169] Exemplarily, the execution subject of this embodiment can be a server, a distributed system, a terminal device, other electronic equipment / computer equipment, or any other device or equipment that can implement the solution of this application, without limitation. Among them, the server can be an independent server, or a server cluster, such as but not limited to any form including a cloud server, a distributed server, a blockchain server, an Internet of Things server, a smart driving platform connected to a car system, etc.

[0170] This embodiment is introduced with the execution subject being a server.

[0171] Exemplarily, the server may first obtain the data to be trained and the training processing task. The data to be trained may include a known data set, for example, a KITTI data set, and / or an autonomously collected data set, etc. This embodiment does not limit the data modality contained in the training data and the method of obtaining the data to be trained. Among them, the introduction of the data modality can refer to the description of the aforementioned target large model embodiment, which will not be repeated here.

[0172] After the server obtains the data to be trained and the training processing tasks, it can input the data to be trained and the training processing tasks into the initial large model.

[0173] Among them, the initial large model in the embodiment of the present application can be understood as the target large model to be trained, for example. The aforementioned target large model embodiment has introduced the framework structure and processing tasks of the target large model, which will not be repeated here. The backbone network (backbone) in the target large model to be trained can be, for example, a fifty-layer residual network (Residual Network 50, referred to as ResNet-50) that has been pre-trained, or a darknet 53 network (Darknet53) that has been pre-trained. The feature extraction unit to be trained in the task network to be trained in the target large model to be trained can be, for example, a neck network (neck), such as a feature pyramid network (Feature Pyramid Network, referred to as FPN), a multi-scale feature fusion network (Multi-scale Feature Fusion Network, referred to as MFFN). The task processing unit to be trained in the task network to be trained in the target large model to be trained can be, for example, a head network (head), such as a target detection head network, a semantic segmentation head network, etc.

[0174] A person skilled in the art can set the network structure of the backbone network, the feature extraction unit to be trained, and the task processing unit to be trained in the target large model to be trained according to actual needs. Furthermore, in the target large model to be trained, the network structure of different backbone networks can be the same, different, or partially the same; the network structure of different feature extraction units to be trained can be the same, different, or partially the same; the network structure of different task processing units to be trained can be the same, different, or partially the same; a person skilled in the art can set it according to actual needs.

[0175] S302, extracting features from the data to be trained through the target backbone network in the initial large model to obtain the first features of the data to be trained; and associating the target backbone network with the data modality of the data to be trained.

[0176] Exemplarily, the target backbone network can be understood as a backbone network associated with the data modality of the data to be trained. After receiving the data to be trained, the target large model to be trained can first determine the data modality contained in the data to be trained by the data format contained in the data to be trained, and then, for each data modality in the data to be trained, determine the backbone network associated with the data modality as the target backbone network, and then, use the target backbone network to perform feature extraction on the data of the data modality in the training data to obtain the first feature of the data of the data modality in the training data. In the process of training the target large model to be trained, when different backbone networks extract the first feature of the data of different data modalities in the training data, the backbone networks do not affect each other.

[0177] S303: Based on the first feature, determine the task processing result of the training data through the feature extraction unit and the task processing unit connected in sequence in the target task network. The target task network is connected to the target backbone network, and the target task network is a task network associated with the training processing task, and different task networks connected to the backbone network are associated with different processing tasks.

[0178] Exemplarily, the target task network may be understood as a task network that is connected to the target backbone network and is used to obtain a task processing result of a training processing task.

[0179] For each training processing task, after obtaining the first feature processed by the target backbone network connected to the target task network associated with the training processing task, the first feature is processed under the training processing task by the feature extraction unit and the task processing unit connected in sequence in the target task network to obtain the task processing result of the training data under the training processing task. In the process of training the target large model to be trained, when the target task networks associated with each training processing task perform training task processing on the first feature, the target task networks do not affect each other.

[0180] S304: Adjust the parameters of the feature extraction unit and the task processing unit in the target task network based on the task processing result.

[0181] Exemplarily, for each training processing task, after obtaining the task processing result after the target task network associated with the training processing task is processed, the parameters of the feature extraction unit and the task processing unit in the target task network are adjusted based on the task processing result. In the process of training the target large model to be trained, there is no need to adjust the parameters of the backbone network, and when adjusting the parameters in each target task network associated with each training processing task, the target task networks do not affect each other.

[0182] The training method for a large model for multi-task processing provided in the embodiment of the present application includes a plurality of feature extraction units and a task processing unit corresponding to each feature extraction unit in the target large model to be trained (initial large model). Based on the feature extraction unit and the task processing unit corresponding to each task, the training result of each task is obtained. Moreover, in the process of training the target large model to be trained, when different backbone networks extract the first feature of the data of different data modes in the training data, the backbone networks do not affect each other; when the target task network associated with each training processing task performs training task processing on the first feature, the target task networks do not affect each other; when adjusting the parameters in each target task network associated with each training processing task, the target task networks do not affect each other. Therefore, the training method for a large model for multi-task processing provided in the present embodiment can realize parallel training of each training processing task. At the same time, in the process of training the target large model to be trained, there is no need to adjust the parameters of the backbone network, so the number of parameters to be updated for training is small. Furthermore, the training speed of each task is fast and efficient.

[0183] Since the target large model to be trained may include multiple backbone networks, and the target backbone network is associated with the data modality of the data to be trained, this embodiment can realize multi-task processing training of the data to be trained containing multiple data modalities at the same time.

[0184] Since the training processing task can be one or more tasks in the target large model to be trained, and each task is independent of each other, it is possible to execute only part of the tasks in the target large model to be trained on the training data, thereby improving the flexibility of model training.

[0185] In a feasible implementation, step S303 includes the following process:

[0186] The first step of step S303: extract the first feature through the feature extraction unit in the target task network to obtain the second feature of the data to be trained. The target task network is connected to the target backbone network, and the target task network is a task network associated with the training processing task, and different task networks connected to the backbone network are associated with different processing tasks.

[0187] Exemplarily, the target task network may be understood as a task network that is connected to the target backbone network and is used to obtain a task processing result of a training processing task.

[0188] For each training processing task, after obtaining the first feature processed by the target backbone network connected to the target task network associated with the training processing task, the first feature is extracted by the feature extraction unit in the target task network to obtain the second feature of the data to be trained under the training processing task. In the process of training the target large model to be trained, when the feature extraction units in the target task network associated with each training processing task extract the second feature from the first feature, the feature extraction units do not affect each other.

[0189] The second step of step S303: based on the task processing unit connected to the feature extraction unit in the target task network, perform the training processing task on the second feature to obtain the task processing result of the data to be trained.

[0190] Exemplarily, for each training processing task, after obtaining the second feature processed by the feature extraction unit in the target task network associated with the training processing task, the training processing task is performed on the second feature by the task processing unit in the target task network to obtain the task processing result of the to-be-trained data under the training processing task. In the process of training the target large model to be trained, when the task processing units in the target task network associated with each training processing task perform the training processing task on the second feature to obtain the task processing result, the task processing units do not affect each other.

[0191] In a feasible implementation, step S304 includes the following process:

[0192] The first step of step S304 is to obtain the loss value of the loss function according to the calibrated task processing results of the training data and the task processing results output by the target task network, wherein the training data has calibrated task processing results.

[0193] Exemplarily, the calibrated task processing results can be understood as the standard task processing results under each training processing task included in the training data. The calibrated task processing results can be obtained from a known data set, such as the KITTI data set; or from an autonomously collected data set. This embodiment does not limit the method for obtaining the calibrated task processing results in the training data.

[0194] For each training processing task, after obtaining the task processing result processed by the task processing unit in the target task network associated with the training processing task, the calibrated task processing result of the training data under the training processing task associated with the target task network and the task processing result output by the target task network are input into the loss function connected to the task processing unit in the target task network to obtain the loss value of the loss function corresponding to the training processing task. Among them, the loss value of the loss function corresponding to the training processing task indicates the difference information between the trained task processing result under the training processing task and the calibrated task processing result under the training processing task. In the process of training the target large model to be trained, when each training processing task determines the loss value of the loss function according to the task processing result and the calibrated task processing result, each training processing task does not affect each other.

[0195] The second step of step S304: according to the loss value of the loss function, update the parameters of the feature extraction unit and the task processing unit in the target task network.

[0196] Exemplarily, for each training processing task, after obtaining the loss value of the loss function corresponding to the training processing task, the loss value of the loss function is sequentially passed through the task processing unit and the feature extraction unit in the target task network associated with the training processing task, and back propagation processing is performed, and the parameters of the task processing unit and the parameters of the feature extraction unit in the target task network are updated in sequence. In the process of training the target large model to be trained, when each training processing task updates the parameters in the target task network according to the loss value of the loss function, the training processing tasks do not affect each other.

[0197] In a possible implementation, for each training processing task, if the loss value of the loss function corresponding to the training processing task has not reached convergence, data enhancement processing can be performed on the training data to update the training data. For example, for the video data to be trained, the data enhancement processing can be, for example, time expansion, space expansion, color space transformation, noise injection, mixed image, random erasure, perspective transformation, etc. of the video data to be trained. Then, repeat the execution Figure 3 The above steps in the embodiment are performed until the loss value of the loss function corresponding to the training processing task reaches convergence.

[0198] In one possible implementation, Figure 3After the training process of the above steps of the embodiment obtains the trained target large model, the modified target large model to be trained can be obtained by adding (or opening) one or more backbone networks and their connected task networks in the trained target large model, and / or modifying one or more backbone networks, and / or deleting (or closing) one or more backbone networks and their connected task networks, and / or, for each backbone network, adding (or opening) one or more task networks connected to the backbone network, and / or, for each backbone network, modifying one or more task networks connected to the backbone network, and / or, for each backbone network, deleting (or closing) one or more task networks connected to the backbone network.

[0199] Furthermore, it is only necessary to add (or open) and / or modify the backbone network and its connected task network in the modified target large model to be trained; and / or add (or open) and / or modify the task network and its connected backbone network, according to Figure 3 The above steps of the embodiment are retrained to obtain the latest trained target large model. In the modified target large model to be trained, when the task network associated with the training processing task and the backbone network connected thereto are unchanged, the processing task does not need to be retrained.

[0200] As an embodiment, it can be combined with Figure 1 , understand that the present application provides a training method for a target large model with only one backbone network, including:

[0201] S11, input the training data and training processing tasks into Figure 1 The target large model to be trained is 10.

[0202] Assume that the backbone network 101 in the target large model 10 to be trained is associated with the video modality. The data to be trained includes the video data to be trained.

[0203] S12, extracting features of the data to be trained through the target backbone network in the target large model 10 to be trained, and obtaining the first feature of the data to be trained.

[0204] For example, after receiving the data to be trained, the target large model 10 to be trained first determines the data mode of the video contained in the data to be trained by the data format contained in the data to be trained. Then, for the video data mode in the data to be trained, the target backbone network is determined to be Figure 1 The backbone network 101 in the Figure 1 The backbone network 101 in the example extracts features from the video data to be trained to obtain the first features of the video data to be trained.

[0205] S13. Extract the first feature through the feature extraction unit in the target task network to obtain the second feature of the data to be trained.

[0206] Exemplarily, referring to the example in step S12, and assuming that the training processing task includes an object detection task for the video data to be trained, the associated target task network is Figure 1 The training processing task also includes the task of regional segmentation of the video data to be trained, and the associated target task network is Figure 1 Task network 102-2 in .

[0207] For the target detection task of the video data to be trained, the example in step S12 is obtained as Figure 1 The task network 102-1 is connected to Figure 1 After the first feature is processed by the backbone network 101, Figure 1 In the task network 102-1 Figure 1 The feature extraction unit 102-1-1 in the embodiment performs feature extraction on the first feature to obtain a second feature of the video data to be trained under the target detection task.

[0208] At the same time, for the region segmentation task of the video data to be trained, the example in step S12 is obtained as Figure 1 The task network 102-2 is connected to Figure 1 After the first feature is processed by the backbone network 101, Figure 1 Task Network 102-2 in Figure 1 The feature extraction unit 102-2-1 in the embodiment extracts the first feature to obtain the second feature of the video data to be trained under the region segmentation task.

[0209] S14. Based on the task processing unit connected to the feature extraction unit in the target task network, a training processing task is performed on the second feature to obtain a task processing result of the data to be trained.

[0210] Exemplarily, referring to the example in step S13, for the target detection task of the video data to be trained, the example in step S13 is obtained Figure 1 In the task network 102-1 Figure 1 After the second feature is processed by the feature extraction unit 102-1-1 in Figure 1 In the task network 102-1 Figure 1 The task processing unit 102-1-2 in the training process performs the target detection task on the second feature to obtain the trained vehicle category and its corresponding target detection frame, the pedestrian category and its corresponding target detection frame, etc. in the video data to be trained.

[0211] At the same time, for the region segmentation task of the video data to be trained, the example in step S13 is obtained as Figure 1 Task Network 102-2 in Figure 1 After the second feature is processed by the feature extraction unit 102-2-1 in Figure 1 Task Network 102-2 in Figure 1 The task processing unit 102-2-2 in the image processing unit performs the region segmentation task on the second feature to obtain the trained conventional road surface categories and their corresponding polygons, field categories and their corresponding polygons, sidewalk pavement and their corresponding polygons, building surfaces and their corresponding polygons, etc. in the video data to be trained.

[0212] S15, obtaining a loss value of a loss function according to the calibrated task processing result of the training data and the task processing result output by the target task network, wherein the training data has a calibrated task processing result.

[0213] For example, referring to the example in step S14, for the target detection task of the video data to be trained, after obtaining the target detection task associated with the target detection task Figure 1 After the task processing results output by the task network 102-1 in the training video data (the trained vehicle categories and their corresponding target detection frames, the pedestrian categories and their corresponding target detection frames, etc. in the video data to be trained), the calibrated task processing results under the target detection task in the video data to be trained (for example, the standard vehicle categories and their corresponding target detection frames, the pedestrian categories and their corresponding target detection frames, etc. in the video data to be trained) are processed, Figure 1 The task processing result output by the task network 102-1 in the Figure 1 The loss function of the task processing unit 102-1-2 connected in the task network 102-1 (the loss function is Figure 1 (not shown) to obtain the loss value of the loss function corresponding to the target detection task.

[0214] At the same time, for the regional segmentation task of the video data to be trained, after obtaining the region segmentation task associated with the task Figure 1 After the task processing results output by the task network 102-2 in the training video data (the trained conventional road surface categories and their corresponding polygons, field categories and their corresponding polygons, sidewalk pavement and their corresponding polygons, building surfaces and their corresponding polygons, etc. in the video data to be trained), the calibrated task processing results under the regional segmentation task in the video data to be trained (for example, the standard conventional road surface categories and their corresponding polygons, field categories and their corresponding polygons, sidewalk pavement and their corresponding polygons, building surfaces and their corresponding polygons, etc. in the video data to be trained) are processed, Figure 1 The task processing result output by the task network 102-2 in the Figure 1 The loss function of the task processing unit 102-2-2 connected in the task network 102-2 (the loss function is Figure 1 (not shown) to obtain the loss value of the loss function corresponding to the region segmentation task.

[0215] S16. Update the parameters of the feature extraction unit and the task processing unit in the target task network according to the loss value of the loss function.

[0216] For example, referring to the example in step S15, for the target detection task of the video data to be trained, after obtaining the loss value of the loss function corresponding to the target detection task (the loss value of the loss function is not in Figure 1 After that, the loss value of the loss function is sequentially passed through the target detection task associated Figure 1 The task processing unit 102-1-2 and the feature extraction unit 102-1-1 in the task network 102-1 perform back propagation processing and update Figure 1 The parameters of the task processing unit 102-1-2 in the task network 102-1 and the feature extraction unit 102-1-1 in the task network 102-1 are calculated. If the loss value of the loss function corresponding to the target detection task has not reached convergence, data enhancement processing is performed on the video data to be trained to update the video data to be trained. Then, steps S11-S16 are repeated until the loss value of the loss function corresponding to the target detection task reaches convergence.

[0217] At the same time, for the region segmentation task of the video data to be trained, after obtaining the loss value of the loss function corresponding to the region segmentation task (the loss value of the loss function is not in Figure 1 After that, the loss value of the loss function is sequentially passed through the region segmentation task associated with Figure 1 The task processing unit 102-2-2 in the task network 102-2 and the feature extraction unit 102-2-1 perform back propagation processing and update Figure 1 The task processing unit 102-2-2 in the task network 102-2 and the parameters of the feature extraction unit 102-2-1 in the task network 102-2 are calculated. If the loss value of the loss function corresponding to the region segmentation task has not reached convergence, data enhancement processing is performed on the video data to be trained to update the video data to be trained. Then, steps S11-S16 are repeated until the loss value of the loss function corresponding to the region segmentation task reaches convergence.

[0218] In a feasible implementation, after the training process of steps S11-S16 is performed to obtain the trained target large model 10, one or more task networks connected to the backbone network can be added (or enabled) in the trained target large model 10 (for example, Figure 1 Add a task network to the trained target model 10). At this time, only the added task network ( Figure 1 The networks for other tasks do not need to be retrained.

[0219] And / or, after the training process of steps S11-S16 is performed to obtain the trained target large model 10, one or more task networks connected to the backbone network can be modified in the trained target large model 10 (for example, Figure 1 In the trained target large model 10, the task network 102-1 is modified. At this time, only the modified task network needs to be retrained, and other task networks do not need to be retrained.

[0220] And / or, after the training process of steps S11-S16 to obtain the trained target large model 10, one or more task networks connected to the backbone network can be deleted (or closed) in the trained target large model 10 (for example, Figure 1 In the trained target large model 10, the task network 102-3 is deleted. At this time, other task networks do not need to be retrained.

[0221] As an embodiment, it can be combined with Figure 2 , understand that the present application provides a method for training a target large model with multiple backbone networks, including:

[0222] S21, input the training data and training processing tasks into Figure 2 The target large model to be trained is 20.

[0223] Assume that the target large model to be trained is 20, Figure 2 (a) shows the backbone network 201 associated with the video mode, Figure 2 (b) shows the backbone network 203 associated with the point cloud data modality. The data to be trained includes video data to be trained and point cloud data to be trained.

[0224] S22, extracting features of the data to be trained through the target backbone network in the target large model 20 to be trained, and obtaining the first features of the data to be trained.

[0225] Exemplarily, after receiving the data to be trained, the target large model 20 to be trained first determines the data modality of the video and the data modality of the point cloud data contained in the data to be trained through the data format contained in the data to be trained.

[0226] Then, according to the video data modality in the training data, the target backbone network is determined as Figure 2 (a) shows the backbone network 201 in FIG. Figure 2 (a) shows that the backbone network 201 extracts features from the video data to be trained to obtain the first features of the video data to be trained.

[0227] At the same time, according to the point cloud data mode in the training data, the target backbone network is determined as Figure 2 (b) shows the backbone network 203 in FIG. Figure 2 (b) shows that the backbone network 203 performs feature extraction on the point cloud data to be trained to obtain the first feature of the point cloud data to be trained.

[0228] S23. Extract the first feature through the feature extraction unit in the target task network to obtain the second feature of the data to be trained.

[0229] Exemplarily, referring to the example in step S22, and assuming that the training processing task includes an object detection task for the video data to be trained, the associated target task network is Figure 2 (a) shows the task network 202-1 in FIG. 1 ; the training processing task also includes the task of segmenting the region of the video data to be trained, and the associated target task network is Figure 2 (a) shows the task network 202-2 in FIG. 1 ; the training processing task also includes the semantic segmentation task of the lane line of the training point cloud data, and the associated target task network is Figure 2 (b) shows the task network 204-1 in FIG. 1 ; the training processing task also includes the key point detection task for the point cloud data to be trained, and the associated target task network is Figure 2 (b) shows the task network 204-3.

[0230] For the target detection task of the video data to be trained, the example in step S22 is obtained as Figure 2 (a) shows the connection of the task network 202-1. Figure 2 (a) shows the first feature after the backbone network 201 processes the first feature. Figure 2 (a) shows the task network 202-1 in FIG. Figure 2 (a) shows the feature extraction unit 202-1-1, which extracts the first feature to obtain the second feature of the video data to be trained under the target detection task.

[0231] At the same time, for the region segmentation task of the video data to be trained, the example in step S22 is obtained as Figure 2(a) shows the connection of the task network 202-2. Figure 2 (a) shows the first feature after the backbone network 201 processes the first feature. Figure 2 (a) shows the task network 202-2 in FIG. Figure 2 (a) shows the feature extraction unit 202-2-1, which extracts the first feature to obtain the second feature of the video data to be trained under the region segmentation task.

[0232] At the same time, for the semantic segmentation task of the lane line of the point cloud data to be trained, the example in step S22 is obtained as Figure 2 (b) shows the connection of the task network 204-1. Figure 2 (b) shows the first feature after the backbone network 203 processes the first feature. Figure 2 (b) shows the task network 204-1 in FIG. Figure 2 (b) shows the feature extraction unit 204-1-1, which performs feature extraction on the first feature to obtain the second feature of the point cloud data to be trained in the semantic segmentation task of the lane line.

[0233] At the same time, for the key point detection task of the point cloud data to be trained, the example in step S22 is obtained as Figure 2 (b) shows the connection of the task network 204-3. Figure 2 (b) shows the first feature after the backbone network 203 processes the first feature. Figure 2 (b) shows the task network 204-3 in FIG. Figure 2 (b) shows the feature extraction unit 204-3-1, which extracts the first feature to obtain the second feature of the point cloud data to be trained under the key point detection task.

[0234] S24. Based on the task processing unit connected to the feature extraction unit in the target task network, a training processing task is performed on the second feature to obtain a task processing result of the data to be trained.

[0235] Exemplarily, referring to the example in step S23, for the target detection task of the video data to be trained, the example in step S23 is obtained Figure 2 (a) shows the task network 202-1 in FIG. Figure 2 (a) shows the second feature after being processed by the feature extraction unit 202-1-1. Figure 2 (a) shows the task network 202-1 in FIG. Figure 2The task processing unit 202-1-2 shown in (a) performs the target detection task on the second feature to obtain the trained vehicle category and its corresponding target detection frame, pedestrian category and its corresponding target detection frame, etc. in the video data to be trained.

[0236] At the same time, for the region segmentation task of the video data to be trained, the example in step S23 obtains Figure 2 (a) shows the task network 202-2 in FIG. Figure 2 (a) shows the second feature after being processed by the feature extraction unit 202-2-1. Figure 2 (a) shows the task network 202-2 in FIG. Figure 2 The task processing unit 202-2-2 shown in (a) performs the region segmentation task on the second feature to obtain the trained conventional road surface categories and their corresponding polygons, field categories and their corresponding polygons, sidewalk pavement and their corresponding polygons, building surfaces and their corresponding polygons, etc. in the video data to be trained.

[0237] At the same time, for the semantic segmentation task of the lane line of the point cloud data to be trained, in the example in step S23, Figure 2 (b) shows the task network 204-1 in FIG. Figure 2 (b) shows the second feature after being processed by the feature extraction unit 204-1-1. Figure 2 (b) shows the task network 204-1 in FIG. Figure 2 (b) shows the task processing unit 204-1-2, which performs the semantic segmentation task of the lane line on the second feature to obtain the trained lane line category and its corresponding polygon in the point cloud data to be trained.

[0238] At the same time, for the key point detection task of the point cloud data to be trained, the example in step S23 is obtained Figure 2 (b) shows the task network 204-3 in FIG. Figure 2 (b) shows the second feature after being processed by the feature extraction unit 204-3-1. Figure 2 (b) shows the task network 204-3 in FIG. Figure 2 (b) illustrates the task processing unit 204-3-2, which performs the key point detection task on the second feature to obtain the trained road intersection points, lane change points, road curvature change points, road slope change points, etc. in the point cloud data to be trained.

[0239] S25, obtaining a loss value of a loss function according to the calibrated task processing result of the training data and the task processing result output by the target task network, wherein the training data has a calibrated task processing result.

[0240] For example, referring to the example in step S24, for the target detection task of the video data to be trained, after obtaining the target detection task associated with the target detection task Figure 2 After the task processing results (the trained vehicle categories and their corresponding target detection frames, the pedestrian categories and their corresponding target detection frames, etc. in the video data to be trained) output by the task network 202-1 are shown in (a), the calibrated task processing results under the target detection task in the video data to be trained (for example, the standard vehicle categories and their corresponding target detection frames, the pedestrian categories and their corresponding target detection frames, etc. in the video data to be trained) are processed. Figure 2 (a) shows that the task processing result output by the task network 202-1 is input to the Figure 2 (a) shows the loss function of the connection between the task processing unit 202-1-2 in the task network 202-1 (the loss function is Figure 2 (not shown) to obtain the loss value of the loss function corresponding to the target detection task.

[0241] At the same time, for the regional segmentation task of the video data to be trained, after obtaining the region segmentation task associated with the task Figure 2 After the task processing results output by the task network 202-2 are shown in (a) (the trained conventional road surface categories and their corresponding polygons, field categories and their corresponding polygons, sidewalk pavement and their corresponding polygons, building surfaces and their corresponding polygons, etc. in the video data to be trained), the calibrated task processing results under the regional segmentation task in the video data to be trained (for example, the standard conventional road surface categories and their corresponding polygons, field categories and their corresponding polygons, sidewalk pavement and their corresponding polygons, building surfaces and their corresponding polygons, etc. in the video data to be trained) are processed, Figure 2 (a) shows that the task processing result output by the task network 202-2 is input to the Figure 2 (a) shows the loss function of the task processing unit 202-2-2 in the task network 202-2 (the loss function is Figure 2 (not shown) to obtain the loss value of the loss function corresponding to the region segmentation task.

[0242] At the same time, for the semantic segmentation task of the lane line of the point cloud data to be trained, after obtaining the semantic segmentation task associated with the lane line Figure 2After the task processing result (the trained lane line category and its corresponding polygon, etc. in the point cloud data to be trained) output by the task network 204-1 is shown in (b), the task processing result of the calibration task under the semantic segmentation task of the lane line in the point cloud data to be trained (for example, the standard lane line category and its corresponding polygon, etc. in the point cloud data to be trained) is processed. Figure 2 (b) shows that the task processing result output by the task network 204-1 is input to the Figure 2 (b) shows the loss function of the connection between the task processing unit 204-1-2 in the task network 204-1 (the loss function is Figure 2 (not shown) to obtain the loss value of the loss function corresponding to the target detection task.

[0243] At the same time, for the key point detection task of the video data to be trained, after obtaining the key point detection task associated with the task Figure 2 After the task processing results (trained road intersections, lane change points, road curvature change points, road slope change points, etc. in the point cloud data to be trained) output by the task network 204-3 are shown in (b), the task processing results of the calibration under the key point detection task in the point cloud data to be trained (for example, standard road intersections, lane change points, road curvature change points, road slope change points, etc. in the point cloud data to be trained) are processed. Figure 2 (b) shows that the task processing result output by the task network 204-3 is input to the Figure 2 (b) shows the loss function of the task processing unit 204-3-2 in the task network 204-3 (the loss function is Figure 2 (not shown) to obtain the loss value of the loss function corresponding to the region segmentation task.

[0244] S26. Update the parameters of the feature extraction unit and the task processing unit in the target task network according to the loss value of the loss function.

[0245] For example, referring to the example in step S25, for the target detection task of the video data to be trained, after obtaining the loss value of the loss function corresponding to the target detection task (the loss value of the loss function is not in Figure 2 After that, the loss value of the loss function is sequentially passed through the target detection task associated Figure 2 (a) shows that the task processing unit 202-1-2 and the feature extraction unit 202-1-1 in the task network 202-1 perform back propagation processing and update Figure 2(a) shows the parameters of the task processing unit 202-1-2 and the feature extraction unit 202-1-1 in the task network 202-1. If the loss value of the loss function corresponding to the target detection task has not reached convergence, the video data to be trained is subjected to data enhancement processing to update the video data to be trained. Then, steps S401-S406 are repeatedly executed until the loss value of the loss function corresponding to the target detection task reaches convergence.

[0246] At the same time, for the region segmentation task of the video data to be trained, after obtaining the loss value of the loss function corresponding to the region segmentation task (the loss value of the loss function is not in Figure 2 After that, the loss value of the loss function is sequentially passed through the region segmentation task associated with Figure 2 (a) shows that the task processing unit 202-2-2 and the feature extraction unit 202-2-1 in the task network 202-2 perform back propagation processing and update Figure 2 (a) shows the parameters of the task processing unit 202-2-2 and the feature extraction unit 202-2-1 in the task network 202-2. If the loss value of the loss function corresponding to the region segmentation task has not reached convergence, data enhancement processing is performed on the video data to be trained to update the video data to be trained. Then, steps S401-S406 are repeated until the loss value of the loss function corresponding to the region segmentation task reaches convergence.

[0247] At the same time, for the semantic segmentation task of the lane line of the point cloud data to be trained, after obtaining the loss value of the loss function corresponding to the semantic segmentation task of the lane line (the loss value of the loss function is not in Figure 2 After that, the loss value of the loss function is sequentially passed through the semantic segmentation task associated with the lane line. Figure 2 (b) shows that the task processing unit 204-1-2 and the feature extraction unit 204-1-1 in the task network 204-1 perform back propagation processing and update Figure 2 (b) shows the parameters of the task processing unit 204-1-2 and the feature extraction unit 204-1-1 in the task network 204-1. If the loss value of the loss function corresponding to the semantic segmentation task of the lane line has not reached convergence, data enhancement processing is performed on the point cloud data to be trained to update the point cloud data to be trained. Then, steps S21-S26 are repeated until the loss value of the loss function corresponding to the semantic segmentation task of the lane line reaches convergence.

[0248] At the same time, for the key point detection task of the point cloud data to be trained, after obtaining the loss value of the loss function corresponding to the key point detection task (the loss value of the loss function is not in Figure 2 After that, the loss value of the loss function is sequentially passed through the key point detection task associated Figure 2 (b) shows that the task processing unit 204-3-2 and the feature extraction unit 204-3-1 in the task network 204-3 perform back propagation processing and update Figure 2 (b) shows the parameters of the task processing unit 204-3-2 and the feature extraction unit 204-3-1 in the task network 204-3. If the loss value of the loss function corresponding to the key point detection task has not reached convergence, the point cloud data to be trained is subjected to data enhancement processing to update the point cloud data to be trained. Then, steps S21 to S26 are repeatedly executed until the loss value of the loss function corresponding to the key point detection task reaches convergence.

[0249] In a feasible implementation, after the training process of steps S21-S26 is performed to obtain the trained target large model 20, one or more backbone networks and their connected task networks (for example, Figure 1 Add the content to Figure 2 In the target large model 20), at this time, only the task network connected to the increased backbone network ( Figure 2 The task networks connected to other backbone networks do not need to be retrained.

[0250] And / or, after the training process of steps S21-S26 is performed to obtain the trained target large model 20, one or more backbone networks may be modified in the trained target large model 20 (for example, modifying Figure 2 (b) shows the backbone network 203 in FIG. 1 . At this time, only the task network connected to the modified backbone network needs to be retrained, and the task networks connected to other backbone networks do not need to be retrained.

[0251] And / or, after the training process of steps S21-S26 to obtain the trained target large model 20, one or more backbone networks and their connected task networks (for example, in Figure 2 In the target large model, delete Figure 2 (b)), at this time, the task networks connected to other backbone networks do not need to be retrained.

[0252] And / or, after the training process of steps S21-S26, the trained target large model 20 is obtained, one or more task networks can be added (or enabled) in the trained target large model 20 (for example, Figure 2 In the target large model, Figure 2(a) shows that a task network is added. At this time, only the added (or enabled) task network needs to be retrained, and other task networks do not need to be retrained. The added (or enabled) task network can be connected to the same backbone network or to different backbone networks.

[0253] And / or, after the training process of steps S21-S26, the trained target large model 20 is obtained, one or more task networks can be modified in the trained target large model 20 (for example, Figure 2 In the target large model, modify Figure 2 (a) shows the task network 202-1 in FIG. 1 . At this time, only the modified task network needs to be retrained, and other task networks do not need to be retrained. The modified task network can be connected to the same backbone network or to different backbone networks.

[0254] And / or, after the training process of steps S21-S26 to obtain the trained target large model 20, one or more task networks (for example, Figure 2 In the target large model, Figure 2 (a) shows that task network 202-3 is deleted. At this time, there is no need to retrain other task networks. The deleted task networks can be connected to the same backbone network or to different backbone networks.

[0255] The training method for a large model for multi-task processing provided by the embodiment of the present application, in the process of training the target large model (initial large model) to be trained, when the feature extraction unit in the target task network associated with each training processing task extracts the second feature from the first feature, the feature extraction units do not affect each other; when the task processing unit in the target task network associated with each training processing task executes the training processing task on the second feature to obtain the task processing result, the task processing units do not affect each other; when each training processing task determines the loss value of the loss function according to the task processing result and the calibrated task processing result, the training processing tasks do not affect each other; when each training processing task updates the parameters in the target task network according to the loss value of the loss function, the training processing tasks do not affect each other; and when the loss value of the loss function corresponding to each training processing task reaches convergence, the training is terminated, without considering whether the loss value of the loss function corresponding to other tasks reaches convergence. Therefore, the training method for a large model for multi-task processing provided by the present embodiment can realize parallel training between each training processing task, and thus the training speed of each processing task is fast and efficient.

[0256] Since each training processing task does not need to consider whether the loss value of the loss function corresponding to other tasks has reached convergence, it is possible to avoid overfitting of the training of each task. When the trained target large model to be trained is used for multi-task processing, the processing results of each task are more accurate.

[0257] Since in the target large model to be trained, the task processing units in the target task network associated with each training processing task have corresponding feature extraction units for extracting the second feature required by the task processing unit, and thus the parameters of the feature extraction unit under each training processing task obtained after the training process are better, therefore, each task can achieve more accurate task processing results when applying the trained target large model.

[0258] In the trained target large model to be trained, the modified target large model to be trained can be obtained by adding (or opening), and / or deleting (or closing) the backbone network and the task network connected thereto; and / or modifying the backbone network; and / or adding (or opening), and / or modifying, and / or deleting (or closing) the task network. Then, only the newly added (or opened), and / or modified parts need to be retrained to obtain the latest trained target large model. By deleting (or closing) the backbone network and the task network connected thereto and / or deleting (or closing) the task network, the training of the processing tasks associated with the task network can be shielded. Moreover, in the modified target large model to be trained, when the task network associated with the processing task and the backbone network connected thereto are not changed, the processing task does not need to be retrained. As a result, the coupling degree of the training of each processing task is low, which improves the scalability and compatibility of the training.

[0259] Figure 4 A flowchart of a multi-task processing method based on a large model provided in this application is as follows: Figure 4 As shown, the method includes:

[0260] S401, input the data to be processed and the target processing task into the target large model.

[0261] Exemplarily, the execution subject of this embodiment can be a server, a distributed system, a terminal device, other electronic equipment / computer equipment, or any other device or equipment that can implement the solution of this application, without limitation. Among them, the server can be an independent server, or a server cluster, such as but not limited to any form including a cloud server, a distributed server, a blockchain server, an Internet of Things server, a smart driving platform connected to a car system, etc.

[0262] This embodiment is introduced with the execution subject being a server.

[0263] For example, the server may first obtain the data to be processed and the target processing task. Then, the data to be processed and the target processing task may be input into the target large model. Figure 3 The target large model to be trained after the method of the embodiment is trained.

[0264] S402, extracting features from the data to be processed through the target backbone network in the target large model to obtain a first feature of the data to be processed; and associating the target backbone network with the data modality of the data to be processed.

[0265] Exemplarily, the target backbone network can be understood as a backbone network associated with the data modality of the data to be processed. After receiving the data to be processed, the target large model can first determine the data modality contained in the data to be processed by the data format contained in the data to be processed, and then, for each data modality in the data to be processed, determine the backbone network associated with the data modality as the target backbone network, and then, use the target backbone network to perform feature extraction on the data of the data modality in the data to be processed to obtain the first feature of the data of the data modality in the data to be processed. In the process of multi-task processing based on the target large model, when different backbone networks perform feature extraction on the data of different data modalities in the data to be processed, the backbone networks do not affect each other.

[0266] S403, extracting the first feature through the feature extraction unit in the target task network to obtain the second feature of the data to be processed; the target task network is connected to the target backbone network, and the target task network is a task network associated with the target processing task, and different task networks connected to the backbone network are associated with different processing tasks.

[0267] Exemplarily, the target task network may be understood as a task network connected to the target backbone network and used to obtain a task processing result of a target processing task.

[0268] For each target processing task, after obtaining the first feature processed by the target backbone network connected to the target task network associated with the target processing task, the feature extraction unit in the target task network extracts the first feature to obtain the second feature of the data to be processed under the target processing task. In the process of multi-task processing based on the target large model, when the feature extraction unit in the target task network associated with each target processing task extracts the second feature from the first feature, the feature extraction units do not affect each other.

[0269] S404: Execute the target processing task on the second feature through the task processing unit in the target task network to obtain the task processing result of the data to be processed.

[0270] Exemplarily, for each target processing task, after obtaining the second feature processed by the feature extraction unit in the target task network associated with the target processing task, the target processing task is executed on the second feature by the task processing unit in the target task network to obtain the task processing result of the to-be-processed data under the target processing task. In the process of multi-task processing based on the target large model, when the task processing units in the target task network associated with each target processing task execute the target processing task on the second feature to obtain the task processing result, the task processing units do not affect each other.

[0271] The multi-task processing method based on a large model provided in an embodiment of the present application inputs the data to be processed and the target processing task into the target large model, and performs the following processing through the target large model: extracts features of the data to be processed through the target backbone network in the target large model to obtain a first feature of the data to be processed; associates the target backbone network with the data modality of the data to be processed; then, extracts features of the first feature through the feature extraction unit in the target task network to obtain a second feature of the data to be processed; the target task network is connected to the target backbone network, and the target task network is a task network associated with the target processing task, and different task networks connected to the backbone network are associated with different processing tasks; and executes the target processing task on the second feature through the task processing unit in the target task network to obtain a task processing result of the data to be processed.

[0272] Since the target large model includes multiple feature extraction units and task processing units corresponding to each feature extraction unit. Based on the feature extraction units and task processing units corresponding to each task, each task processing result is obtained. Moreover, in the process of multi-task processing based on the target large model, when different backbone networks extract the first feature of data of different data modes in the processed data, the backbone networks do not affect each other; when the feature extraction units in the target task network associated with each target processing task extract the second feature from the first feature, the feature extraction units do not affect each other. When the task processing units in the target task network associated with each target processing task execute the target processing task on the second feature to obtain the task processing result, the task processing units do not affect each other. Therefore, this embodiment can provide a multi-task parallel processing method based on a large model. Furthermore, each task is processed quickly and efficiently.

[0273] Since a plurality of backbone networks may be included in the target large model, and the target backbone network is associated with the data modality of the data to be processed, this embodiment can realize multi-task processing of the data to be processed of multiple data modalities at the same time.

[0274] Since in the target large model, each target task network associated with each target processing task has a feature extraction unit for extracting the second feature required by the task processing unit in the target task network, the semantic feature information required by the task processing unit can be extracted more accurately, thereby obtaining a higher accuracy rate of the task processing result.

[0275] Since the target processing task can be one or more tasks in the target large model, and each task is independent of each other, it is possible to execute only part of the tasks in the target large model for the data to be processed, thereby improving the flexibility of model application.

[0276] In a feasible implementation, before extracting the first feature through the feature extraction unit in the target task network to obtain the second feature of the data to be processed, if it is determined that the available task network connected to the target backbone network does not include the target task network, a prompt message for the target processing task is generated and issued; wherein the prompt message indicates that the target processing task cannot be processed. The target task network is connected to the target backbone network, and the target task network is a task network associated with the target processing task, and different task networks connected to the backbone network are associated with different processing tasks.

[0277] Exemplarily, the server may respond to an enable instruction and enable the task network indicated by the enable instruction so that the task network under the specified processing task is in an enabled state; wherein the enable instruction includes the specified processing task, and the enable instruction is used to instruct to enable the task network under the specified processing task.

[0278] The server also responds to the closing instruction and closes the task network indicated by the closing instruction to put the task network under the specified processing task in a closed state; wherein the closing instruction includes the specified processing task, and the closing instruction is used to instruct the closing of the task network under the specified processing task.

[0279] If the task network is in the open state, then the task network is available. If the task network is in the closed state, then the task network is unavailable.

[0280] The target task network may be understood as a task network connected to the target backbone network and used to obtain the task processing result of the target processing task.

[0281] For each target processing task, if it is determined that the target task network associated with the target processing task is an unusable task network, a prompt message indicating that the target processing task cannot be processed is generated and issued.

[0282] In a feasible implementation, in a multi-task processing process based on a large model, the modified target large model can be obtained by adding (or opening) one or more backbone networks and their connected task networks in the target large model, and / or modifying one or more backbone networks, and / or deleting (or closing) one or more backbone networks and their connected task networks, and / or, for each backbone network, adding (or opening) one or more task networks connected to the backbone network, and / or, for each backbone network, modifying one or more task networks connected to the backbone network, and / or, for each backbone network, deleting (or closing) one or more task networks connected to the backbone network.

[0283] Furthermore, the newly added (or opened) backbone network and the task network connected to it in the modified target large model can realize the multi-task processing of the data mode associated with the newly added (or opened) backbone network. At the same time, the modified backbone network and the task network connected to it in the modified target large model can realize more accurate multi-task processing of the data mode associated with the modified backbone network. At the same time, the backbone network and the task network connected to it deleted (or closed) in the modified target large model can realize the multi-task processing of the data mode associated with the deleted (or closed) backbone network. At the same time, the added (or opened) task network in the modified target large model can realize the processing of the processing task associated with the task network for the data to be processed. At the same time, the added (or opened) task network in the modified target large model can realize more accurate processing of the processing task associated with the task network for the data to be processed. At the same time, the deleted (or closed) task network in the modified target large model can realize the shielding of the processing of the processing task associated with the task network for the data to be processed.

[0284] Furthermore, in the modified target macro model, when the task network associated with the processing task and the backbone network connected thereto are not changed, the task processing process of the processing task is not affected.

[0285] After obtaining the task processing results of each task, the server can make decisions based on these task processing results. For example, the server can control the vehicle to turn and avoid pedestrians based on the target detection box of pedestrians in the task processing results.

[0286] As an embodiment, it can be combined with Figure 1 , understand that the present application provides a multi-task processing method based on a target large model with only one backbone network, including:

[0287] S31, input the data to be processed and the target processing task into the target large model 10.

[0288] Assume that the backbone network 101 in the target large model 10 is associated with the video modality. The data to be processed includes the video data to be processed.

[0289] S32. Extract features of the data to be processed through the target backbone network in the target large model 10 to obtain the first feature of the data to be processed.

[0290] Exemplarily, after receiving the data to be processed, the target large model 10 first determines the data mode of the video contained in the data to be processed by the data format contained in the data to be processed. Then, for the video data mode in the data to be processed, the target backbone network is determined to be Figure 1 The backbone network 101 in the Figure 1 The backbone network 101 in the example extracts features from the video data to be processed to obtain a first feature of the video data to be processed.

[0291] S33: If it is determined that the available task networks connected to the target backbone network do not include the target task network, a prompt message for the target processing task is generated and issued.

[0292] Exemplarily, referring to the example in step S32, and assuming that the target processing task includes a target detection task for the video data to be processed, the associated target task network is Figure 1 The target processing task also includes a task of segmenting the video data to be processed, and the associated target task network is Figure 1 The task network 102-2 in (assumed to be an unusable task network).

[0293] Then, for the region segmentation task of the video data to be processed, determine the associated Figure 1 If the task network 102-2 in the example is an unusable task network, a prompt message is generated and issued that the region segmentation processing cannot be performed on the video data to be processed.

[0294] S34: If it is determined that the target task network associated with the target processing task is an available task network, a feature extraction unit in the target task network is used to extract the first feature to obtain a second feature of the data to be processed.

[0295] For example, referring to the example in step S33, for the target detection task of the video data to be processed, the associated processing task is determined. Figure 1 The task network 102-1 in is an available task network, then the example in step S32 is obtained with Figure 1 The task network 102-1 is connected to Figure 1 After the first feature is processed by the backbone network 101, Figure 1 In the task network 102-1 Figure 1 The feature extraction unit 102-1-1 in the embodiment performs feature extraction on the first feature to obtain a second feature of the video data to be processed under the target detection task.

[0296] S35. After obtaining the second feature of the data to be processed, the task processing unit in the target task network executes the target processing task on the second feature to obtain the task processing result of the data to be processed.

[0297] Exemplarily, referring to the example in step S34, for the target detection task of the video data to be processed, the example in step S34 is obtained Figure 1 In the task network 102-1 Figure 1 After the second feature is processed by the feature extraction unit 102-1-1 in Figure 1 In the task network 102-1 Figure 1 The task processing unit 102-1-2 in the processing performs the target detection task on the second feature to obtain the category of the vehicle and its corresponding target detection frame, the category of the pedestrian and its corresponding target detection frame, etc. in the video data to be processed.

[0298] In a feasible implementation, when the target large model 10 is used, one or more task networks connected to the backbone network can be added (or opened) in the target large model 10 (for example, in Figure 1 In the target large model 10, a task network is added. At this time, based on the added (or opened) task network ( Figure 1 (not shown), the data of the data modality associated with the backbone network to which it is connected is processed by the task network associated with the task processing, and does not affect the task processing of other task networks.

[0299] And / or, when the target large model 10 is used, one or more task networks connected to the backbone network can be modified in the target large model 10 (for example, in Figure 1 In the target large model 10, the task network 102-1 is modified. At this time, based on the modified task network, the data of the data modality associated with the backbone network connected to it can be re-processed for the processing tasks associated with the task network without affecting the task processing of other task networks.

[0300] And / or, when the target large model 10 is used, one or more task networks connected to the backbone network can be deleted (or closed) in the target large model 10 (for example, in Figure 1 In the target large model 10, the task network 102-3 is deleted. At this time, the task processing of other task networks is not affected.

[0301] As an embodiment, it can be combined with Figure 2 , understand that the present application provides a multi-task processing method based on a target large model with multiple backbone networks, including:

[0302] S41, input the data to be processed and the target processing task into the target large model 20.

[0303] Assume that in the target macro model 20, Figure 2 (a) shows the backbone network 201 associated with the video mode, Figure 2 (b) shows the backbone network 203 associated with the point cloud data mode. The data to be processed includes video data to be processed and point cloud data to be processed.

[0304] S42, extracting features of the data to be processed through the target backbone network in the target large model 10 to obtain the first feature of the data to be processed.

[0305] Exemplarily, after receiving the data to be processed, the target large model 20 first determines the data modality of the video and the data modality of the point cloud data in the data to be processed through the data format contained in the data to be processed.

[0306] Then, according to the video data mode in the data to be processed, the target backbone network is determined as Figure 2 (a) shows the backbone network 201 in FIG. Figure 2 (a) shows that the backbone network 201 extracts features from the video data to be processed to obtain the first features of the video data to be processed.

[0307] At the same time, according to the point cloud data mode in the data to be processed, the target backbone network is determined as Figure 2 (b) shows the backbone network 203 in FIG. Figure 2 (b) shows that the backbone network 203 performs feature extraction on the point cloud data to be processed to obtain the first feature of the point cloud data to be processed.

[0308] S43: If it is determined that the available task networks connected to the target backbone network do not include the target task network, a prompt message for the target processing task is generated and issued.

[0309] Exemplarily, referring to the example in step S42, and assuming that the target processing task includes a target detection task for the video data to be processed, the associated target task network is Figure 2 (a) shows the task network 202-1 (assuming it is a usable task network); the target processing task also includes a region segmentation task for the video data to be processed, and the associated target task network is Figure 2(a) shows the task network 202-2 (assumed to be an unusable task network); the target processing task also includes the semantic segmentation task of the lane line of the point cloud data to be processed, and the associated target task network is Figure 2 (b) shows the task network 204-1 (assuming it is a usable task network); the target processing task also includes a key point detection task for the point cloud data to be processed, and the associated target task network is Figure 2 (b) shows the task network 204 - 3 (assumed to be an unavailable task network).

[0310] Then, for the region segmentation task of the video data to be processed, determine the associated Figure 2 The task network 202 - 2 shown in (a) is an unusable task network, and a prompt message is generated and issued indicating that the region segmentation process cannot be performed on the video data to be processed.

[0311] For the key point detection task of the point cloud data to be processed, determine the Figure 2 The task network 204 - 3 shown in (b) is an unusable task network, and a prompt message is generated and issued indicating that key point detection cannot be performed on the point cloud data to be processed.

[0312] S44: If it is determined that the target task network associated with the target processing task is an available task network, a feature extraction unit in the target task network is used to extract the first feature to obtain a second feature of the data to be processed.

[0313] For example, referring to the example in step S43, for the target detection task of the video data to be processed and the semantic segmentation task of the lane line of the point cloud data to be processed, the processing task associated with the target detection task is determined. Figure 2 (a) shows the task network 202-1 and Figure 2 The task network 204-1 shown in (b) is an available task network, and the example in step S42 is obtained with Figure 2 (a) shows the connection of the task network 202-1. Figure 2 (a) shows the first feature after the backbone network 201 processes the first feature. Figure 2 (a) shows the task network 202-1 in FIG. Figure 2 (a) shows the feature extraction unit 202-1-1, which extracts the first feature to obtain the second feature of the video data to be processed under the target detection task.

[0314] At the same time, for the semantic segmentation task of lane lines of the point cloud data to be processed, the same as Figure 2 (b) shows the connection of the task network 204-1. Figure 2 (b) shows the first feature after the backbone network 203 processes the first feature. Figure 2 (b) shows the task network 204-1 in FIG. Figure 2 (b) shows the feature extraction unit 204-1-1, which performs feature extraction on the first feature to obtain the second feature of the point cloud data to be processed under the semantic segmentation task of the lane line.

[0315] S45. After obtaining the second feature of the data to be processed, the task processing unit in the target task network executes the target processing task on the second feature to obtain the task processing result of the data to be processed.

[0316] Exemplarily, referring to the example in step S44, for the target detection task of the video data to be processed, the example in step S44 is obtained Figure 2 (a) shows the task network 202-1 in FIG. Figure 2 (a) shows the second feature after being processed by the feature extraction unit 202-1-1. Figure 2 (a) shows the task network 202-1 in FIG. Figure 2 (a) shows the task processing unit 202-1-2, which performs the target detection task on the second feature to obtain the vehicle category and its corresponding target detection frame, the pedestrian category and its corresponding target detection frame, etc. in the video data to be processed.

[0317] At the same time, for the semantic segmentation task of the lane line of the point cloud data to be processed, in the example in step S44, Figure 2 (b) shows the task network 204-1 in FIG. Figure 2 (b) shows the second feature after being processed by the feature extraction unit 204-1-1. Figure 2 (b) shows the task network 204-1 in FIG. Figure 2 (b) illustrates the task processing unit 204-1-2, which performs the semantic segmentation task of the lane line on the second feature to obtain the category of the lane line and its corresponding polygon in the point cloud data to be processed.

[0318] In a feasible implementation, when the target large model 20 is used, one or more backbone networks and their connected task networks (for example, Figure 1 Add the content to Figure 2 In the target large model 20), at this time, the task network ( Figure 2(not shown), multi-task processing is performed on the data of the data modality associated with the backbone network without affecting the task processing of the task networks connected to other backbone networks.

[0319] And / or, when the target large model 20 is used, one or more backbone networks may be modified in the target large model 20 (for example, modifying Figure 2 (b) shows the backbone network 203), at this time, based on the modified backbone network and the task network connected to it, the data of the data modality associated with the backbone network can be re-multitasked without affecting the task processing of the task networks connected to other backbone networks.

[0320] And / or, when the target large model 20 is used, one or more backbone networks and their connected task networks (for example, in Figure 2 In the target large model, delete Figure 2 (b) indicates the content), at this time, it does not affect the task processing of the task network connected to other backbone networks.

[0321] And / or, when the target large model 20 is used, one or more task networks can be added (or opened) in the target large model 20 (for example, in Figure 2 In the target large model, Figure 2 (a) shows that a task network is added. At this time, the task network can be added (or opened) based on the task network ( Figure 2 (not shown), the task network performs task processing on the data of the data mode associated with the backbone network to which it is connected, and does not affect the task processing of other task networks. Among them, the added (or enabled) task network can be connected to the same backbone network or to different backbone networks.

[0322] And / or, when the target large model 20 is used, one or more task networks may be modified in the target large model 20 (e.g., Figure 2 In the target large model, modify Figure 2 (a) shows the task network 202-1). At this time, based on the modified task network, the data of the data modality associated with the backbone network to which it is connected can be re-processed for the processing tasks associated with the task network without affecting the task processing of other task networks. The modified task network can be connected to the same backbone network or to different backbone networks.

[0323] And / or, when the target large model 20 is used, one or more task networks may be deleted (or closed) in the target large model 20 (for example, in Figure 2 In the target large model, Figure 2(a) shows that task network 202-3 is deleted, and at this time, the task processing of other task networks is not affected. The deleted task network can be connected to the same backbone network or to different backbone networks.

[0324] In the multi-task processing method based on a large model provided in the embodiment of the present application, if the target task network is not included in the available task network connected to the target backbone network, a prompt message indicating that the target processing task associated with the target task network cannot be processed can be generated and issued. In this way, each processing task can be monitored and the unprocessable tasks can be intervened in time.

[0325] In the multi-task processing process based on the large model, by adding (or opening), and / or deleting (or closing) the backbone network and the task network connected thereto in the target large model; and / or modifying the backbone network; and / or adding (or opening), and / or modifying, and / or deleting (or closing) the task network, a modified target large model is obtained. It is possible to realize multi-task processing of the data modality associated with the newly added (or opened) backbone network based on the modified target large model, more accurately realize multi-task processing of the data modality associated with the modified backbone network, shield the multi-task processing of the data modality associated with the deleted (or closed) backbone network, realize the processing of the processing task associated with the task network added (or opened) to the processing data, more accurately realize the processing of the processing task associated with the task network modified to the processing data, shield the processing of the processing task associated with the task network deleted (or closed) to the processing data. Moreover, in the modified target large model, when the task network associated with the processing task and the backbone network connected thereto are not changed, the task processing process of the processing task is not affected. Therefore, the coupling degree of each task processing on the data to be processed is low, which improves the scalability and compatibility of task processing.

[0326] Figure 5 The schematic diagram of the structure of the multi-task processing device based on the large model provided in this application is as follows: Figure 5 As shown, the multi-task processing device 50 based on the large model provided in this embodiment includes:

[0327] An input unit 501 is used to input the data to be processed and the target processing task into the target large model, and execute the processing process of the following units through the target large model;

[0328] The first extraction unit 502 is used to extract features of the data to be processed through the target backbone network in the target large model to obtain the first feature of the data to be processed; the target backbone network is associated with the data mode of the data to be processed;

[0329] The second extraction unit 503 is used to extract the first feature through the feature extraction unit in the target task network to obtain the second feature of the data to be processed; the target task network is connected to the target backbone network, and the target task network is a task network associated with the target processing task, and different task networks connected to the backbone network are associated with different processing tasks;

[0330] The execution unit 504 is used to execute the target processing task on the second feature through the task processing unit in the target task network to obtain the task processing result of the data to be processed.

[0331] The multi-task processing device based on a large model provided in this embodiment can execute the method provided in the above-mentioned multi-task processing method embodiment based on a large model. Its implementation principle and technical effects are similar, and will not be described in detail in this embodiment.

[0332] Figure 6 A schematic diagram of a large model training device for multi-task processing provided in this application is shown in FIG. Figure 6 As shown, the training device 60 for a large model of multi-task processing provided in this embodiment includes:

[0333] An input unit 601 is used to input the training data and the training processing task into the initial large model, and execute the processing process of the following units through the initial large model;

[0334] An extraction unit 602 is used to extract features from the data to be trained through the target backbone network in the initial large model to obtain a first feature of the data to be trained; the target backbone network is associated with the data modality of the data to be trained;

[0335] A determination unit 603 is used to determine the task processing result of the to-be-trained data based on the first feature through the feature extraction unit and the task processing unit connected in sequence in the target task network; the target task network is connected to the target backbone network, and the target task network is a task network associated with the training processing task, and different task networks connected to the backbone network are associated with different processing tasks;

[0336] The adjustment unit 604 is used to adjust the parameters of the feature extraction unit and the task processing unit in the target task network based on the task processing result.

[0337] The training device for a large model for multi-task processing provided in this embodiment can execute the training method for a large model for multi-task processing provided in the above method embodiment. Its implementation principle and technical effects are similar, and are not described in detail in this embodiment.

[0338] Figure 7 A schematic diagram of another large model training device for multi-task processing provided by the present application is shown in FIG. Figure 7As shown, the training device 70 for a large model of multi-task processing provided in this embodiment includes:

[0339] An input unit 701 is used to input the training data and the training processing task into the initial large model, and execute the processing process of the following units through the initial large model;

[0340] An extraction unit 702 is used to extract features from the data to be trained through the target backbone network in the initial large model to obtain a first feature of the data to be trained; the target backbone network is associated with the data mode of the data to be trained;

[0341] A determination unit 703 is used to determine the task processing result of the to-be-trained data based on the first feature through the feature extraction unit and the task processing unit connected in sequence in the target task network; the target task network is connected to the target backbone network, and the target task network is a task network associated with the training processing task, and different task networks connected to the backbone network are associated with different processing tasks;

[0342] The adjustment unit 704 is used to adjust the parameters of the feature extraction unit and the task processing unit in the target task network based on the task processing result.

[0343] In one possible implementation, the initial large model includes M backbone networks, and different backbone networks are associated with different data modalities; wherein the backbone network is connected to N task networks, and different task networks connected to the backbone network are associated with different processing tasks; the task network includes a feature extraction unit and a task processing unit connected in sequence; M is an integer greater than 0; and N is an integer greater than 0.

[0344] In a possible implementation manner, the determining unit 703 is specifically configured to:

[0345] An extraction module 7031 is used to extract the first feature through a feature extraction unit in the target task network to obtain a second feature of the data to be trained;

[0346] The execution module 7032 is used to perform the training processing task on the second feature based on the task processing unit connected to the feature extraction unit in the target task network, and obtain the task processing result of the data to be trained.

[0347] In a possible implementation, the data to be trained has calibrated task processing results;

[0348] In a possible implementation manner, the adjusting unit 704 is specifically configured to:

[0349] A determination module 7041 is used to obtain a loss value of a loss function according to a calibrated task processing result of the training data and a task processing result output by the target task network;

[0350] The updating module 7042 is used to update the parameters of the feature extraction unit and the task processing unit in the target task network according to the loss value of the loss function.

[0351] In a possible implementation, the data modality includes at least one of image, video, audio, document, text, and point cloud data.

[0352] In a possible implementation, when the data modality includes images or videos, the processing task includes at least one of target recognition, target classification, event recognition, behavior recognition, gesture recognition, emotion recognition, target segmentation, and data format conversion;

[0353] When the data modality includes audio, the processing tasks include at least one of speech recognition, voice dialogue, voice inquiry, data format conversion, speech emotion recognition, speech-to-text conversion, and speech summary generation;

[0354] When the data modality includes documents, the processing tasks include at least one of document classification, key information extraction, document summary generation, document sentiment analysis, and document security detection;

[0355] When the data modality includes text, the processing tasks include at least one of text classification, entity recognition, text summary generation, text translation, text deduplication, text sentiment analysis, and format error recognition;

[0356] When the data modality includes point cloud data, the processing tasks include at least one of target recognition, target segmentation, point cloud denoising, and point cloud completion.

[0357] The training device for a large model for multi-task processing provided in this embodiment can execute the training method for a large model for multi-task processing provided in the above method embodiment. Its implementation principle and technical effects are similar, and are not described in detail in this embodiment.

[0358] Figure 8 This is a schematic diagram of the structure of the electronic device provided in this application. Figure 8 As shown, the electronic device 80 provided in this embodiment includes: at least one processor 801 and a memory 802. Optionally, the device 100 further includes a communication component 803. The processor 801, the memory 802 and the communication component 803 are connected via a bus 804.

[0359] In a specific implementation process, at least one processor 801 executes the computer-executable instructions stored in the memory 802, so that at least one processor 801 executes the above method.

[0360] The specific implementation process of the processor 801 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0361] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0362] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0363] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0364] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0365] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0366] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0367] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0368] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0369] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0370] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0371] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0372] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

[0373] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A multi-task processing method based on a large model, characterized in that: The method comprises: The data to be processed and the target processing task are input into the target large model, and the following processing is performed through the target large model: Extracting features of the data to be processed by the target backbone network in the target large model to obtain a first feature of the data to be processed; associating the target backbone network with the data modality of the data to be processed; The first feature is extracted by a feature extraction unit in a target task network to obtain a second feature of the data to be processed; the target task network is connected to the target backbone network, and the target task network is a task network associated with the target processing task, and different task networks connected to the backbone network are associated with different processing tasks; and The target processing task is executed on the second feature by a task processing unit in the target task network to obtain a task processing result of the data to be processed.

2. The method according to claim 1, characterized in that The target large model includes M backbone networks, and different backbone networks are associated with different data modalities; wherein the backbone network is connected to N task networks, and different task networks connected to the backbone network are associated with different processing tasks; the task network includes a feature extraction unit and a task processing unit connected in sequence; M is an integer greater than 0; and N is an integer greater than 0.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: If it is determined that the target task network is not included in the available task networks connected to the target backbone network, prompt information for the target processing task is generated and issued; wherein the prompt information indicates that the target processing task cannot be processed.

4. The method according to claim 1 or 2, characterized in that: The data modality includes at least one of image, video, audio, document, text, and point cloud data.

5. The method according to claim 4, characterized in that When the data modality includes images or videos, the processing task includes at least one of target recognition, target classification, event recognition, behavior recognition, posture recognition, emotion recognition, target segmentation, key point detection, and data format conversion; When the data modality includes audio, the processing task includes at least one of speech recognition, voice dialogue, voice inquiry, data format conversion, speech emotion recognition, speech-to-text conversion, and speech summary generation; When the data modality includes documents, the processing task includes at least one of document classification, key information extraction, document summary generation, document sentiment analysis, and document security detection; When the data modality includes text, the processing task includes at least one of text classification, entity recognition, generating text summaries, text translation, text deduplication, text sentiment analysis, and format error recognition; When the data modality includes point cloud data, the processing task includes at least one of target recognition, target segmentation, key point detection, point cloud denoising, and point cloud completion.

6. A target large model for multi-tasking processing, characterized in that The target large model includes: M backbone networks, different backbone networks are associated with different data modalities; the backbone networks are used to extract the first feature of the to-be-processed data of the associated data modality; The backbone network is connected to N task networks, and different task networks connected to the backbone network are associated with different processing tasks; the task network includes a feature extraction unit and a task processing unit connected in sequence; the feature extraction unit is used to extract a first feature extracted by the connected backbone network to obtain a second feature; the task processing unit is used to perform an associated processing task on the second feature extracted by the connected feature extraction unit to obtain a task processing result of the data to be processed; Wherein, the M is an integer greater than 0, and the N is an integer greater than 0.

7. The target macromodel according to claim 6, characterized in that: The feature extraction unit includes a multi-scale feature extraction module and a first fusion module; The multi-scale feature extraction module is used to extract the first feature extracted by the connected backbone network to obtain data features in multiple dimensions; The first fusion module is used to perform feature fusion on the data features in the multiple dimensions to obtain the second features.

8. The target macromodel according to claim 6, characterized in that: The feature extraction unit includes an up-sampling module, a down-sampling module, and a second fusion module; The upsampling module is used to perform high-level feature extraction on the first feature extracted by the connected backbone network to obtain high-level semantic feature information; The downsampling module is used to perform low-level feature extraction on the first feature extracted by the connected backbone network to obtain low-level semantic feature information; The second fusion module is used to perform feature fusion on the high-level semantic feature information and the low-level semantic feature information to obtain a second feature.

9. The target macromodel according to any one of claims 6 to 8, characterized in that: The data modality includes at least one of image, video, audio, document, text, and point cloud data.

10. The target macromodel according to claim 9, characterized in that: When the data modality includes images or videos, the processing task includes at least one of target recognition, target classification, event recognition, behavior recognition, posture recognition, emotion recognition, target segmentation, key point detection, and data format conversion; When the data modality includes audio, the processing task includes at least one of speech recognition, voice dialogue, voice inquiry, data format conversion, speech emotion recognition, speech-to-text conversion, and speech summary generation; When the data modality includes documents, the processing task includes at least one of document classification, key information extraction, document summary generation, document sentiment analysis, and document security detection; When the data modality includes text, the processing task includes at least one of text classification, entity recognition, generating text summaries, text translation, text deduplication, text sentiment analysis, and format error recognition; When the data modality includes point cloud data, the processing task includes at least one of target recognition, target segmentation, key point detection, point cloud denoising, and point cloud completion.

11. A training method for a large model for multi-task processing, characterized in that: include: The data to be trained and the training processing tasks are input into the initial large model, and the following processing is performed by the initial large model: Extracting features of the data to be trained by the target backbone network in the initial large model to obtain a first feature of the data to be trained; associating the target backbone network with the data modality of the data to be trained; Based on the first feature, the task processing result of the data to be trained is determined by the feature extraction unit and the task processing unit connected in sequence in the target task network, and the parameters of the feature extraction unit and the task processing unit in the target task network are adjusted based on the task processing result; the target task network is connected to the target backbone network, and the target task network is the task network associated with the training processing task, and different task networks connected to the backbone network are associated with different processing tasks.

12. The method according to claim 11, characterized in that The initial large model includes M backbone networks, and different backbone networks are associated with different data modalities; wherein the backbone network is connected to N task networks, and different task networks connected to the backbone network are associated with different processing tasks; the task network includes a feature extraction unit and a task processing unit connected in sequence; M is an integer greater than 0; and N is an integer greater than 0.

13. The method according to claim 11, characterized in that Based on the first feature, determining the task processing result of the to-be-trained data through the feature extraction unit and the task processing unit sequentially connected in the target task network includes: Extracting the first feature by a feature extraction unit in the target task network to obtain a second feature of the data to be trained; Based on the task processing unit connected to the feature extraction unit in the target task network, the training processing task is performed on the second feature to obtain the task processing result of the data to be trained.

14. The method according to claim 11, characterized in that The data to be trained has a calibrated task processing result; Adjusting parameters of a feature extraction unit and a task processing unit in the target task network based on the task processing result includes: Obtaining a loss value of a loss function according to the calibrated task processing result of the to-be-trained data and the task processing result output by the target task network; According to the loss value of the loss function, the parameters of the feature extraction unit and the task processing unit in the target task network are updated.

15. The method according to any one of claims 11 to 14, characterized in that The data modality includes at least one of image, video, audio, document, text, and point cloud data.

16. The method according to claim 15, characterized in that When the data modality includes images or videos, the processing task includes at least one of target recognition, target classification, event recognition, behavior recognition, posture recognition, emotion recognition, target segmentation, key point detection, and data format conversion; When the data modality includes audio, the processing task includes at least one of speech recognition, voice dialogue, voice inquiry, data format conversion, speech emotion recognition, speech-to-text conversion, and speech summary generation; When the data modality includes documents, the processing task includes at least one of document classification, key information extraction, document summary generation, document sentiment analysis, and document security detection; When the data modality includes text, the processing task includes at least one of text classification, entity recognition, generating text summaries, text translation, text deduplication, text sentiment analysis, and format error recognition; When the data modality includes point cloud data, the processing task includes at least one of target recognition, target segmentation, key point detection, point cloud denoising, and point cloud completion.

17. An electronic device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor executes the method described in any one of claims 1-5 or 11-16, or the processor executes the processing process of the target large model described in any one of claims 6-10.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method described in any one of claims 1-5 or 11-16, or to implement the processing process of the target large model described in any one of claims 6-10.

19. A computer program product, comprising a computer program, which, when executed by a processor, implements the method described in any one of claims 1 to 5 or 11 to 16, or implements the processing process of the target large model described in any one of claims 6 to 10.

Images