Multi-perception task processing method and device based on model merging and electronic equipment

Through the model-based merging method, the combination matrix and separation matrix are used to merge multiple perceptual task adaptation models into target-aware task processing model, solving the problem of time-consuming and space-consuming multi-perceptual task inference in the existing technology, and achieving efficient Multitasking and memory savings.

CN119027786APending Publication Date: 2024-11-26TSINGHUA UNIVERSITY +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410909544.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-08
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

In the prior art, implementing multi-perception task inference requires a lot of time overhead and memory space, making it difficult to efficiently process multi-data sets and multiple perception tasks.

Method used

Using a model-based merging method, by constructing a merge matrix and a separation matrix, multiple perceptual task adaptation models are merged into a target-aware task processing model, and task processing is used to achieve simultaneous processing of multi-tasks.

Benefits of technology

It speeds up the inference speed, saves memory space, and expands the application scope and processing capabilities of the model, effectively dealing with multi-datasets and multi-aware tasks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027786A_ABST
    Figure CN119027786A_ABST
Patent Text Reader

Abstract

The invention provides a multi-perception task processing method and device based on model merging and electronic equipment. The method comprises the steps that multiple perception task adaptation models to be merged are determined; on the basis of the constructed merging matrix and separation matrix, merging the multiple sensing task adaptation models to obtain a target sensing task processing model; and based on the target sensing task processing model, processing one or more sensing tasks to obtain a task processing result. According to the method, multiple sensing task adaptive models are merged into the same target sensing task processing model through the constructed merging matrix and separation matrix, so that the characteristics of the original multiple sensing task models are reserved, and the target sensing task processing model is allowed to be more effective in multi-data-set and multi-sensing-task processing; the application range and the processing capacity of the model are expanded, and the storage space of the multi-data-set and multi-perception-task adaptation model is effectively saved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a multi-sensory task processing method, device and electronic equipment based on model merging. Background Art

[0002] Most of the existing technologies are based on a single data set and a single perception task application of a single model. If you want to apply multiple data sets and multiple perception tasks at the same time, you need to use the models corresponding to each data set and each perception task for reasoning at the same time, which requires a lot of time and memory space.

[0003] Therefore, it is necessary to solve the problem that implementing multi-sensory task reasoning in the existing technology requires a lot of time and memory space. Summary of the invention

[0004] The present invention provides a multi-sensory task processing method, device and electronic device based on model merging, which are used to overcome the defect that a large amount of time and memory space are required to realize multi-sensory task reasoning in the prior art. It can not only speed up the reasoning speed, but also save memory space.

[0005] On the one hand, the present invention provides a multi-perception task processing method based on model merging, including: determining multiple perception task adaptation models to be merged; merging the multiple perception task adaptation models based on a constructed merging matrix and a separation matrix to obtain a target perception task processing model; based on the target perception task processing model, processing one or more perception tasks to obtain a task processing result.

[0006] Furthermore, the multiple perception task adaptation models are merged based on the constructed merging matrix and separation matrix, including: constructing a merging matrix and a separation matrix corresponding to different network layers in the target perception task processing model; determining a merging weight matrix according to the merging matrix and the separation matrix, and original weight matrices of the multiple perception task adaptation models; and applying the merging weight matrix to the merged model to obtain the target perception task processing model.

[0007] Furthermore, the multiple perception task adaptation models are merged based on the constructed merging matrix and separation matrix, including: merging a set part of the network layers of the multiple perception task adaptation models, and the other network layers other than the set part of the network layers retain the independent network layer structures of the multiple perception task adaptation models.

[0008] Furthermore, the construction of merging matrices and separation matrices corresponding to different network layers in the target perception task processing model includes: obtaining an image to be processed, and based on a plurality of perception task adaptation models, obtaining a plurality of feature vectors corresponding to the image to be processed; splicing a plurality of feature vectors, and according to the correlation of all feature pairs in the spliced ​​feature vectors, iteratively selecting feature pairs with the highest correlation for merging to obtain a merged feature vector; determining a mapping matrix that maps a plurality of feature vectors to the merged feature vector as the merging matrix; and determining the pseudo-inverse of the merging matrix as the separation matrix.

[0009] Further, according to the merging matrix and the separation matrix, and the original weight matrices of the multiple perception task adaptation models, a merging weight matrix is ​​determined, including: determining the merging weight matrix by setting a formula, and the setting formula is as follows:

[0010] in, represents the merge weight matrix, represents the merge matrix, represents the original weight matrix from the perceptual task adaptation model A, represents the original weight matrix from the perceptual task adaptation model B, , denote the parts of the separation matrix, Represents the concatenation of feature vectors.

[0011] Furthermore, based on the target perception task processing model, one or more perception tasks are processed to obtain task processing results, including: acquiring an image to be processed; performing matrix multiplication operations on the image to be processed using a merged weight matrix of the target perception task processing model to obtain task processing results corresponding to one or more perception tasks.

[0012] Furthermore, the perception task adaptation model includes a target detection model, a target tracking model, a classification model and a semantic segmentation model.

[0013] In the second aspect, the present invention also provides a multi-perception task processing device based on model merging, including: a multi-perception task adaptation model determination module, used to determine the multiple perception task adaptation models to be merged; a multi-perception task adaptation model merging module, used to merge the multiple perception task adaptation models based on the constructed merging matrix and separation matrix to obtain a target perception task processing model; a multi-perception task processing module, used to process one or more perception tasks based on the target perception task processing model to obtain a task processing result.

[0014] In a third aspect, the present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the multi-perception task processing method based on model merging as described above is implemented.

[0015] In a fourth aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the multi-perception task processing methods based on model merging as described above.

[0016] The multi-perception task processing method based on model merging provided by the present invention determines multiple perception task adaptation models to be merged, and merges the multiple perception task adaptation models based on the constructed merging matrix and separation matrix to obtain a target perception task processing model, and then processes one or more perception tasks based on the target perception task processing model to obtain a task processing result. The method merges multiple perception task adaptation models into the same target perception task processing model through the constructed merging matrix and separation matrix, which not only retains the characteristics of the original multiple perception task model, but also allows the target perception task processing model to perform more effectively in multi-dataset and multi-perception task processing, expands the application scope and processing capacity of the model, and effectively saves the storage space of the multi-dataset and multi-perception task adaptation model. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A schematic flow chart of a multi-sensory task processing method based on model merging provided in an embodiment of the present invention is shown.

[0019] Figure 2 It is a schematic diagram of constructing a merging matrix and a separating matrix provided in an embodiment of the present invention.

[0020] Figure 3 It is a schematic diagram of the application of the separation matrix and the merging matrix provided in an embodiment of the present invention.

[0021] Figure 4 It is a schematic diagram of weight merging of multiple perception task adaptation models provided by an embodiment of the present invention.

[0022] Figure 5 It is a schematic diagram of a merging mode provided in an embodiment of the present invention.

[0023] Figure 6 It is a structural diagram of a multi-sensory task processing device based on model merging provided in an embodiment of the present invention.

[0024] Figure 7 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0026] It should be noted that in the existing technology, most of them are based on a single data set and a single perception task application of a single model. If you want to apply multiple data sets and multiple perception tasks at the same time, you need to use the models corresponding to each data and each perception task for reasoning at the same time, which will require a lot of time and memory space (such as storage space for multiple data sets and their corresponding multiple models).

[0027] In view of this, the present invention proposes a multi-sensory task processing method based on model merging. Specifically, Figure 1 A schematic flow chart of a multi-sensory task processing method based on model merging provided in an embodiment of the present invention is shown.

[0028] like Figure 1 As shown, the method includes steps S110-S130, and these steps will be described in detail below.

[0029] S110, determining multiple perception task adaptation models to be merged.

[0030] It can be understood that in this embodiment, if one wants to realize the processing of multiple perceptual tasks while effectively saving the storage space of multiple data sets and multiple perceptual task adaptation models, it is necessary to first determine the multiple perceptual task adaptation models to be merged, and prepare for the subsequent merging of multiple perceptual task adaptation models.

[0031] Specifically, the perception task adaptation model refers to a model adapted for the perception task. The perception task adaptation model corresponds to the perception task and the data set. The number of perception task adaptation models can be set according to actual needs and the category of the perception task, and is not specifically limited here.

[0032] For example, in a specific embodiment, the perception task includes at least two of a target detection task, a target tracking task, a classification task, and a semantic segmentation task, and the perception task adaptation model includes two or a combination of two or more of a target detection model, a target tracking model, a classification model, and a semantic segmentation model.

[0033] It should be noted that the multiple perception task adaptation models to be merged are all trained models that can perform independent reasoning for the corresponding perception tasks. They have relatively independent model parameters and have certain advantages in reasoning for the corresponding perception tasks.

[0034] It should also be noted that the model architecture of the multiple perception task adaptation models can be set according to actual needs and is not specifically limited here.

[0035] For example, in a specific embodiment, multiple perception task adaptation models are constructed based on convolutional neural networks.

[0036] S120, based on the constructed merging matrix and separation matrix, merging the multiple perception task adaptation models to obtain a target perception task processing model.

[0037] It can be understood that, based on the determination of multiple perception task adaptation models to be merged in step S110, further, step S120 will merge the multiple perception task adaptation models into the same model based on the constructed merging matrix and separation matrix to obtain the target perception task processing model.

[0038] Specifically, multiple perception task adaptation models are merged into the target perception task processing model, and corresponding merging matrices and separation matrices are designed for each network layer in the target perception task processing model. When calculating the weight of the current network layer in the target perception task processing model, consider separating the weight layer of the current network layer through the separation matrix. The two or more separated weight layers are respectively used as the input of the network layer of each perception task adaptation model, and corresponding processing is performed. Then, the multiple outputs processed by each perception task adaptation model are merged through the merging matrix as the final output of the current network layer.

[0039] By following the above process, this embodiment solves the network layer parameters of the merged model (target perception task processing model), such as the weight matrix, in closed form. Then, different merging modes can be designed according to the requirements of different perception tasks to obtain the target perception task processing model.

[0040] In other words, the core of integrating multiple perception task adaptation models into the target perception task processing model is to use merging matrices and separating matrices to process model weights and features.

[0041] First, by calculating the correlation of multiple model feature vectors corresponding to multiple perceptual task adaptation models, a merging matrix is ​​constructed to achieve effective integration of feature layers. A separation matrix is ​​applied to the feature vectors integrated by the merging matrix to decompose the integrated feature vectors back to the feature space of multiple perceptual task adaptation models, ensuring that the weights of each perceptual task adaptation model can be correctly applied to the appropriate feature parts.

[0042] Then, at the weight level, the eigenvectors adjusted by the separation matrix are applied to the weights of multiple perception task adaptation models respectively, and then the outputs are integrated through the merging matrix to form the final merged weight matrix. These weight matrices capture the information of multiple perception task adaptation models, so that the merged target perception task processing model can process multiple perception tasks at the same time.

[0043] In this step, the functional fusion of multiple perception task adaptation models is achieved through precise matrix operations, while minimizing performance loss and expanding the application scope and processing capabilities of the model.

[0044] It is worth mentioning that the different merging modes designed above mean that when merging multiple perception task adaptation models, some network layers of the multiple perception task adaptation models can be selected for merging according to actual needs, and the other network layers retain the original network layer structure of the multiple perception task adaptation models.

[0045] S130, based on the target perception task processing model, one or more perception tasks are processed to obtain task processing results.

[0046] It can be understood that in step S120, based on the constructed merging matrix and separation matrix, multiple perception task adaptation models are merged to obtain the target perception task processing model. Further, step S130 will use the target perception task processing model to infer one or more perception tasks to obtain the final task processing result.

[0047] Specifically, the target perception task processing model obtained by merging multiple perception task adaptation models can process multiple perception tasks at the same time. During processing, the image to be processed corresponding to the perception task is input into the target perception task processing model. The merged weight matrix in the target perception task processing model will perform matrix multiplication on the image to be processed, thereby obtaining the corresponding task processing result.

[0048] Among them, the task processing results include but are not limited to target detection results (target detection box), classification results (description of which category the object belongs to), target tracking results (movement trajectory of moving objects over time) and semantic segmentation results (semantic category corresponding to each pixel in the image to be processed).

[0049] In this embodiment, by determining multiple perception task adaptation models to be merged, and based on the constructed merging matrix and separation matrix, merging the multiple perception task adaptation models, a target perception task processing model is obtained, and then based on the target perception task processing model, one or more perception tasks are processed to obtain a task processing result. This method merges multiple perception task adaptation models into the same target perception task processing model through the constructed merging matrix and separation matrix, which not only retains the characteristics of the original multiple perception task models, but also allows the target perception task processing model to perform more effectively in multi-dataset and multi-perception task processing, expands the application scope and processing capabilities of the model, and effectively saves the storage space of the multi-dataset and multi-perception task adaptation model.

[0050] On the basis of the above embodiments, the process of merging multiple perception task adaptation models will be described in detail below.

[0051] Based on the constructed merging matrix and separation matrix, multiple perception task adaptation models are merged, including: constructing merging matrices and separation matrices corresponding to different network layers in the target perception task processing model; determining a merging weight matrix according to the merging matrix and separation matrix, and original weight matrices of multiple perception task adaptation models; applying the merging weight matrix to the merged model to obtain the target perception task processing model.

[0052] In a specific embodiment, the construction process of the merge matrix and the split matrix is ​​described.

[0053] Construct the merging matrix and separation matrix corresponding to different network layers in the multiple perceptual task adaptation models. Specifically, first obtain the image to be processed, and based on the multiple perceptual task adaptation models, obtain multiple feature vectors corresponding to the image to be processed; then, concatenate multiple feature vectors, and according to the correlation of all feature pairs in the concatenated feature vectors, iteratively select the feature pairs with the highest correlation for merging to obtain a merged feature vector; finally, determine the mapping matrix that maps multiple feature vectors to the merged feature vector as the merging matrix, and determine the pseudo-inverse of the merging matrix as the separation matrix.

[0054] Specifically, Figure 2 FIG. 2 shows a schematic diagram of constructing a merging matrix and a separating matrix provided by an embodiment of the present invention. Figure 2 As shown, the construction process of the merge matrix begins with the correlation calculation of two or more model feature vectors. This embodiment takes the merging of two perception task adaptation models A and B as an example. First, the image to be processed is input into the perception task adaptation models A and B, and the feature vector output by the perception task adaptation model A can be obtained. And the feature vector output by the perception task adaptation model B , the eigenvector and are all n-dimensional vectors.

[0055] Then, the feature vector and Concatenate to form a new feature vector ,in, .

[0056] For the concatenated feature vector , calculate the correlation of all feature pairs within it, specifically the correlation coefficient can be used To measure two characteristics and By comparing all feature pairs The correlation of the features is evaluated to determine which feature pairs should be merged. Based on these correlation results, the feature pairs with the highest correlation are selected for merging. This step is implemented by a greedy algorithm, that is, the feature pair with the highest correlation is selected, and then this pair of features is removed from the feature pool, and this process is iterated until all feature pairs are processed.

[0057] The merging process can be represented as constructing a merging matrix , , used to average or interpolate related feature pairs to form a new merged feature vector In other words, the merge matrix is ​​the eigenvector and Mapping to the merged feature vector The mapping matrix.

[0058] Whenever there is a feature and When selected as merge, the merge matrix The values ​​at the corresponding index positions of the corresponding rows in are set to 0.5, and the values ​​at other positions are set to 0.

[0059] The above can get the constructed merge matrix .

[0060] After obtaining the merged feature vector, the next network layer needs to receive feature vectors that are compatible with the original adaptation models of each perception task. To this end, this embodiment designs a separation matrix , which is used to merge the feature vector (i.e., merged feature vector) "Unpack" back to the original feature vector .

[0061] Specifically, the separation matrix is the merge matrix The pseudo-inverse of can usually be expressed as ,in this way, .

[0062] Since the merge matrix The values ​​of the two features are averaged into one value, and this combined value is redistributed to the positions of the two original features, so that the merging operation has as little loss as possible when it is passed through the network.

[0063] It is worth mentioning that the calculation and application of these matrices are the key steps to realize the merging of model functions, so that a single target perception task processing model can handle multiple perception tasks that were originally handled separately by multiple perception task adaptation models. In this way, this embodiment can expand the application scope and processing capabilities of the model without sacrificing too much of the original performance of the model.

[0064] In another specific embodiment, a specific process of determining a merging weight matrix according to a merging matrix and a separating matrix is ​​described in detail.

[0065] It can be understood that the process of determining the merge weight matrix ensures that the features extracted from multiple different perceptual task adaptation models can work together in the new merged model (target perceptual task processing model). This requires not only merging features, but also correctly merging the weight matrices of the network layers from different perceptual task adaptation models.

[0066] Still taking the combination of two perception task adaptation models as an example, Figure 3 A schematic diagram of the application of the separation matrix and the merging matrix provided in the embodiment of the present invention is shown.

[0067] like Figure 3 As shown, first, the constructed separation matrix is ​​applied.

[0068] It is understandable that the merged feature vector needs to be mapped back to the feature space of the original perceptual task adaptation model so that the weight matrix of each perceptual task adaptation model can be applied. This step is done by separating the matrix To achieve this, we merge the feature vectors It is separated into two parts, corresponding to the feature spaces of the original two perception task adaptation models respectively. For details, please refer to equations (1)-(2).

[0069] (1). (2).

[0070] In formulas (1)-(2), represents the feature vector mapped back to the perception task adaptation model A, represents the feature vector mapped back to the perception task adaptation model B, , denote the parts of the separation matrix, represents the merged feature vector.

[0071] Then, the original weight matrix is ​​applied.

[0072] It can be understood that after applying the separation matrix, the two separated feature vectors are multiplied by the original weight matrix of the adaptation model of each perception task, as shown in Equations (3)-(4).

[0073] (3). (4).

[0074] In formulas (3)-(4), represents the output vector of the perception task adaptation model A, represents the output vector of the perception task adaptation model B, represents the original weight matrix from the perceptual task adaptation model A, represents the original weight matrix from the perceptual task adaptation model B.

[0075] Next, the merging matrix is ​​applied.

[0076] It can be understood that the output vectors of the perception task adaptation models A and B are merged by the matrix Merge into the final output feature vector for use by the next network layer of the target perception task processing model. For details, see formula (5).

[0077] (5).

[0078] In formula (5), Represents the output feature vector of the current network layer in the target perception task processing model.

[0079] Finally, the weight matrix is ​​formed.

[0080] It can be understood that after applying the separation matrix, the original weight matrix, and the merger matrix, a new merger weight matrix is ​​formed , which will be used for the corresponding network layer of the target perception task processing model during actual reasoning. The merged weight matrix is ​​obtained by transforming and merging the features of the perception task adaptation model A and the perception task adaptation model B, which contains the information of the two perception task adaptation models.

[0081] Specifically, the merging weight matrix is ​​determined by setting formula (6).

[0082] (6).

[0083] In formula (6), represents the merge weight matrix, represents the merge matrix, represents the original weight matrix from the perceptual task adaptation model A, represents the original weight matrix from the perceptual task adaptation model B, , denote the parts of the separation matrix, and represents the original weight matrix after applying the separation matrix, Represents the concatenation of feature vectors.

[0084] Based on the above process, the weight layers (weight matrices) from two independent perception task adaptation models can be successfully merged into the same single multi-task model. This not only retains the characteristics of the respective perception task adaptation models, but also allows the merged target perception task processing model to perform more effectively in multi-dataset and multi-perception task processing, expanding the application scope and processing capabilities of the model.

[0085] It should be noted that the above only takes the merging of two perception task adaptation models as an example. When there are more than two perception task adaptation models to be merged, the specific process can refer to the above process and will not be expanded here one by one.

[0086] It should also be noted that the above example only illustrates the overall merging process of a single network layer, not the merging process of all network layers.

[0087] Additionally, in some embodiments, Figure 4 A schematic diagram of weight merging of multiple perception task adaptation models provided by an embodiment of the present invention is shown.

[0088] like Figure 4 As shown in the figure, taking the merger of two perception task adaptation models as an example, before the merger, the two perception task adaptation models have their own independent weight layers. and After merging, the target perception task processing model only contains one weight layer .

[0089] Then, the combined weight matrix is ​​applied to the combined model to obtain the target perception task processing model, which can process multiple perception tasks simultaneously.

[0090] In this embodiment, by constructing a merging matrix and a separation matrix corresponding to different network layers in the target perception task processing model, and determining a merging weight matrix based on the merging matrix and the separation matrix, as well as the original weight matrix of the multiple perception task adaptation models, and then applying the merging weight matrix to the merged model, the target perception task processing model is obtained, thereby, based on the target perception task processing model, one or more perception tasks are processed to obtain the task processing result. This method merges multiple perception task adaptation models into the same target perception task processing model through the constructed merging matrix and separation matrix, which not only retains the characteristics of the original multiple perception task models, but also allows the target perception task processing model to perform more effectively in the processing of multiple data sets and multiple perception tasks, expands the application scope and processing capabilities of the model, and effectively saves the storage space of the multiple data sets and multiple perception task adaptation models.

[0091] On the basis of the above embodiments, further based on the constructed merging matrix and separation matrix, multiple perception task adaptation models are merged, including: merging the set part of the network layers of the multiple perception task adaptation models, and retaining the independent network layer structures of the multiple perception task adaptation models in other network layers other than the set part of the network layers.

[0092] It is understandable that in some cases, if the output spaces of two perceptual task adaptation models are incompatible (for example, perceptual task adaptation model A needs to output the semantic segmentation result of an image, and perceptual task adaptation model B needs to output the target detection result of an image), or if completely merging the two perceptual task adaptation models will result in a large loss of accuracy, you can choose to stop merging after a certain network layer and retain the subsequent independent network layer structures.

[0093] In this way, the merging operation of the model occurs in a set part of the network layers instead of all the network layers, thereby reducing the negative impact on the model performance while maintaining a certain merging efficiency. When implementing, you only need to pass the merging matrix and the separation matrix to the specified network layer, and then process the remaining network layers independently.

[0094] The number of partial networks may be set according to actual needs and is not specifically limited here.

[0095] Figure 5 A schematic diagram of a merging mode provided by an embodiment of the present invention is shown, which shows a merging mode in which the first four network layers are merged, and then the merging is stopped to keep the network layers of each model as they are.

[0096] In this embodiment, by merging the set part of network layers of multiple perception task adaptation models and retaining the independent network layer structures of the multiple perception task adaptation models, the negative impact on model performance can be reduced while maintaining a certain merging efficiency.

[0097] On the basis of the above embodiments, further, based on the target perception task processing model, one or more perception tasks are processed to obtain task processing results, including: obtaining an image to be processed; performing matrix multiplication operations on the image to be processed using the merged weight matrix of the target perception task processing model to obtain task processing results corresponding to one or more perception tasks.

[0098] It can be understood that the target perception task processing model obtained by merging multiple perception task adaptation models can process multiple perception tasks at the same time. During processing, the image to be processed corresponding to the perception task is input into the target perception task processing model, and the merged weight matrix in the target perception task processing model will perform matrix multiplication on the image to be processed, thereby obtaining the corresponding task processing result.

[0099] Among them, the task processing results include but are not limited to target detection results (target detection box), classification results (description of which category the object belongs to), target tracking results (movement trajectory of moving objects over time) and semantic segmentation results (semantic category corresponding to each pixel in the image to be processed).

[0100] In this embodiment, by acquiring the image to be processed and using the merged weight matrix of the target perception task processing model to perform matrix multiplication operation on the image to be processed, the task processing results corresponding to one or more perception tasks are obtained, thereby effectively saving the storage space of multiple data sets and multiple perception task adaptation models.

[0101] Corresponding to the multi-sensory task processing method based on model merging described in the above embodiments, the present invention also provides a multi-sensory task processing device based on model merging. Specifically, Figure 6 A schematic diagram of the structure of a multi-sensory task processing device based on model merging provided in an embodiment of the present invention is shown.

[0102] like Figure 6 As shown, the device includes: a multi-perception task adaptation model determination module 610, used to determine a plurality of perception task adaptation models to be merged; a multi-perception task adaptation model merging module 620, used to merge the plurality of perception task adaptation models based on a constructed merging matrix and a separation matrix to obtain a target perception task processing model; a multi-perception task processing module 630, used to process one or more perception tasks based on the target perception task processing model to obtain a task processing result.

[0103] In this embodiment, the multiple perception task adaptation models to be merged are determined by the multi-perception task adaptation model determination module 610, and the multi-perception task adaptation model merging module 620 merges the multiple perception task adaptation models based on the constructed merging matrix and separation matrix to obtain the target perception task processing model, and then the multi-perception task processing module 630 processes one or more perception tasks based on the target perception task processing model to obtain the task processing result. The device merges the multiple perception task adaptation models into the same target perception task processing model through the constructed merging matrix and separation matrix, which not only retains the characteristics of the original multiple perception task models, but also allows the target perception task processing model to perform more effectively in the processing of multiple data sets and multiple perception tasks, expands the application scope and processing capabilities of the model, and effectively saves the storage space of the multiple data sets and multi-perception task adaptation models.

[0104] It should be noted that the multi-sensory task processing device based on model merging provided in this embodiment can correspond to the multi-sensory task processing method based on model merging described in the above embodiments, and will not be repeated here.

[0105] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute a multi-sensory task processing method based on model merging, the method comprising: determining a plurality of perceptual task adaptation models to be merged; merging the plurality of perceptual task adaptation models based on the constructed merging matrix and separation matrix to obtain a target perceptual task processing model; processing one or more perceptual tasks based on the target perceptual task processing model to obtain a task processing result.

[0106] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0107] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, it is implemented to execute the multi-perception task processing method based on model merging provided by the above-mentioned methods. The method includes: determining multiple perception task adaptation models to be merged; merging the multiple perception task adaptation models based on the constructed merging matrix and separation matrix to obtain a target perception task processing model; processing one or more perception tasks based on the target perception task processing model to obtain a task processing result.

[0108] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0109] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0110] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-sensory task processing method based on model merging, It is characterized in that include: Determine the adaptation models for multiple perception tasks to be merged; Based on the constructed merging matrix and separation matrix, the plurality of perception task adaptation models are merged to obtain a target perception task processing model; Based on the target perception task processing model, one or more perception tasks are processed to obtain task processing results.

2. The multi-sensory task processing method based on model merging according to claim 1, It is characterized in that The method of merging the plurality of perception task adaptation models based on the constructed merging matrix and separation matrix includes: Constructing merging matrices and separation matrices corresponding to different network layers in the target perception task processing model; Determining a merging weight matrix according to the merging matrix and the separating matrix, and original weight matrices of the adaptation models of multiple perception tasks; The combined weight matrix is ​​applied to the combined model to obtain the target perception task processing model.

3. The multi-sensory task processing method based on model merging according to claim 1, It is characterized in that The method of merging the plurality of perception task adaptation models based on the constructed merging matrix and separation matrix includes: The set partial network layers of the multiple perception task adaptation models are merged, and the other network layers other than the set partial network layers retain the independent network layer structures of the multiple perception task adaptation models.

4. The multi-sensory task processing method based on model merging according to claim 2, It is characterized in that The step of constructing the merging matrix and the separating matrix corresponding to different network layers in the target perception task processing model includes: Acquire an image to be processed, and based on a plurality of perception task adaptation models, acquire a plurality of feature vectors corresponding to the image to be processed; Concatenate multiple feature vectors, and iteratively select the feature pairs with the highest correlation to merge according to the correlation of all feature pairs in the concatenated feature vectors to obtain a merged feature vector; Determine a mapping matrix that maps a plurality of eigenvectors to the merged eigenvector as the merged matrix; The pseudo-inverse of the merging matrix is ​​determined as the separating matrix.

5. The multi-sensory task processing method based on model merging according to claim 2, It is characterized in that According to the merging matrix and the separation matrix, and the original weight matrices of the multiple perception task adaptation models, a merging weight matrix is ​​determined, including: The combined weight matrix is ​​determined by setting a formula, and the setting formula is as follows: in, represents the merge weight matrix, represents the merge matrix, represents the original weight matrix from the perceptual task adaptation model A, represents the original weight matrix from the perceptual task adaptation model B, , denote the parts of the separation matrix, Represents the concatenation of feature vectors.

6. The multi-sensory task processing method based on model merging according to claim 5, It is characterized in that The processing of one or more perception tasks based on the target perception task processing model to obtain task processing results includes: Get the image to be processed; The merged weight matrix of the target perception task processing model is used to perform matrix multiplication operation on the image to be processed to obtain task processing results corresponding to one or more perception tasks.

7. The multi-sensory task processing method based on model merging according to any one of claims 1 to 6, It is characterized in that The perception task adaptation model includes a target detection model, a target tracking model, a classification model and a semantic segmentation model.

8. A multi-sensory task processing device based on model merging, It is characterized in that include: A multi-perception task adaptation model determination module, used to determine multiple perception task adaptation models to be merged; A multi-perception task adaptation model merging module, used to merge the multiple perception task adaptation models based on the constructed merging matrix and separation matrix to obtain a target perception task processing model; The multi-perception task processing module is used to process one or more perception tasks based on the target perception task processing model to obtain task processing results.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the program, the multi-sensory task processing method based on model merging according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the multi-sensory task processing method based on model merging according to any one of claims 1 to 7 is implemented.