Data processing method, model training method, and related device
By adding task indication information to the input of the machine learning model, the problem of resource waste in multi-task processing is solved, and efficient task execution and optimized resource utilization are achieved.
Patent Information
- Application Number
- PCT/CN2024/095693
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-30
- Filing Date
- 2024-05-28
- Publication Date
- 2024-12-05
AI Technical Summary
In machine learning models, the same data may require multiple tasks to be performed, but not all tasks are required to be performed every time, resulting in wasted computer resources, because many of the generated predictions are redundant.
By adding the first information to the input of the machine learning model, indicating the task to be performed, the model generates only the required prediction results, thereby performing the task adaptively and avoiding redundant calculations.
It effectively avoids the waste of computer resources, improves resource utilization, and improves the flexibility and adaptability of the model.
Smart Images

Figure CN2024095693_05122024_PF_FP_ABST
Abstract
Description
A data processing method, model training method and related equipment
[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on May 30, 2023, with application number 202310627237.1 and invention name “A data processing method, a model training method and related equipment”, the entire contents of which are incorporated by reference into this application. Technical Field
[0002] The present application relates to the field of artificial intelligence, and in particular to a data processing method, a model training method, and related equipment. Background Art
[0003] Artificial intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that seeks to understand the essence of intelligence and develop new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making. Using machine learning models within AI for data processing is a common application.
[0004] In some scenarios, it may be necessary to perform multiple tasks on the same data through a machine learning model at the same time. The current approach is to deploy a machine learning model on the device that can perform N tasks at one time. When it is necessary to perform at least one of the N tasks on a certain data (hereinafter referred to as "first data" for the convenience of description), the first data is input into the machine learning model. By performing N tasks on the first data through the machine learning model, N prediction results corresponding to the N tasks can be obtained, and then at least one prediction result actually required can be obtained from the N prediction results.
[0005] Since not all of the N tasks need to be executed every time the machine learning model is used to process the first data, sometimes only some of the N prediction results generated by the machine learning model are needed, and the other prediction results will be discarded, which can easily cause a waste of computer resources.
[0006] Summary of the Invention
[0007] The embodiments of the present application provide a data processing method, a model training method, and related equipment, in which first information is added to the input of the machine learning model. Based on the first information, the first machine learning model can know which tasks need to be performed on the first data, and then can adaptively perform the required tasks on the first data, avoiding the generation of redundant prediction results, thereby avoiding the waste of computer resources.
[0008] To solve the above technical problems, the embodiments of the present application provide the following technical solutions:
[0009] In a first aspect, an embodiment of the present application provides a data processing method that can be used in a multi-task processing scenario in the field of artificial intelligence, wherein a first machine learning model deployed on an execution device has the ability to simultaneously execute N tasks on input first data, and the method includes: the execution device can obtain first data and first information, and the first information indicates M tasks to be executed on the first data, where N is an integer greater than 1, and M is an integer greater than or equal to 1. The execution device inputs the first data and the first information into the first machine learning model, processes the first data through the first machine learning model, and obtains M prediction results output by the first machine learning model that correspond one-to-one to the M tasks.
[0010] In this implementation, first information is added to the input of the first machine learning model. The first information is used to indicate that at least one task needs to be performed on the first data. The first machine learning model outputs at least one prediction result that corresponds one-to-one with the at least one task. That is, the first machine learning model can know which tasks need to be performed on the first data based on the first information, and then can adaptively perform the required tasks on the first data, avoiding the generation of redundant prediction results, thereby avoiding wasting computer resources.
[0011] In one possible implementation, the first information can be expressed as a first vector, which may include N elements corresponding one-to-one to N tasks. When the value of any one of the N elements (hereinafter referred to as "target element" for the convenience of description) is the first value, it means that a task corresponding to the target element needs to be executed on the first data. When the value of the target element is the second value, it means that the task corresponding to the target element does not need to be executed on the first data; the first value and the second value are different.
[0012] In one possible implementation, a first machine learning model includes multiple neural network layers, including at least one first neural network layer among the multiple neural network layers, wherein an execution device processes first data using the first machine learning model, including: the execution device determining parameters used by each first neural network layer based on first information; optionally, when the input first information is different, the parameters used by the first neural network layer may be different. Since the type of operation performed by each first neural network layer is preset, after the execution device determines the parameters used by any one neural network layer, the first neural network layer may be used to process the second data based on the parameters used by the aforementioned first neural network layer to obtain a processing result of the second data generated by the first neural network layer, where the second data is the first data or feature information of the first data. For example, if any one of the first neural network layers (hereinafter referred to as the "target neural network layer" for convenience of description) is the first neural network layer used by the first machine learning model to process the first data, the second data may be the first data; if the target neural network layer is not the first neural network layer used by the first machine learning model to process the first data, the second data may be feature information of the first data.
[0013] In this implementation, when the number of parameters adopted by the first machine learning model is larger, more computer resources are consumed when using the first machine learning model for data processing, and more abundant information can be mined from the input first data; correspondingly, when the number of parameters adopted by the first machine learning model is smaller, fewer computer resources are consumed when using the first machine learning model for data processing, and the information mined from the input first data will be relatively less; the first machine learning model has the ability to execute N tasks simultaneously. When all of the N tasks are executed by the first machine learning model, each first neural network layer can adopt more parameters, but not all of the N tasks are necessarily executed each time by the first machine learning model (that is, the M tasks that need to be executed on the first data may be part of the N tasks). Determining the parameters adopted by each first neural network layer based on the first information is conducive to achieving adaptation between the parameters adopted by each first neural network layer and the "M tasks that need to be executed on the first data" to avoid wasting computer resources.
[0014] In one possible implementation, the multiple first neural network layers included in the first machine learning model can all be deployed in the feature extraction network of the first machine learning model. If the second data is the feature information of the first data, the processing result of the second data is the updated feature information of the first data; if the second data is the first data, the processing result of the second data is the feature information of the first data.
[0015] In this implementation, since the feature extraction network in the first machine learning model consumes more computer resources in the process of processing the first data using the first machine learning model, adjusting the parameters used by the neural network layer in the feature extraction network based on the first information is conducive to significantly reducing the waste of computer resources.
[0016] In one possible implementation, the execution device determines the parameters used by the first neural network layer based on the first information, including: the execution device obtains the first parameters corresponding to the first neural network layer; determines the second parameters corresponding to the first neural network layer based on the feature information of the first information; and the execution device determines the parameters used by the first neural network layer based on the first parameters and the second parameters. Exemplarily, the "first parameters corresponding to the target neural network layer (i.e., any one of the first neural network layers)" can be understood as parameters that are independent of the task, that is, the "first parameters corresponding to the target neural network layer" are used as a set of bases. Regardless of which tasks among the N tasks are included in the M tasks, the aforementioned "first parameters corresponding to the target neural network layer" will be obtained; the "second parameters corresponding to the target neural network layer" can be understood as parameters related to the task. Therefore, when the first information is different (i.e., when the M tasks executed on the first data are different), the second parameters corresponding to the target neural network layer can be different.
[0017] In this implementation, the parameters adopted by each first neural network layer are decoupled into task-independent parameters (i.e., first parameters) and task-related parameters (i.e., second parameters) corresponding to the first neural network layer. After determining which tasks need to be performed on the input first data, the task-independent parameters corresponding to the first neural network layer can be determined based on the feature information of the first information, and then the parameters finally adopted by each first neural network layer are determined based on the task-independent parameters and the task-related parameters. Since the first information will affect the second parameters corresponding to each first neural network layer, it is beneficial to achieve adaptability between the parameters adopted by each first neural network layer and the first information. In addition, no matter which tasks need to be performed on the input first data, the first parameters corresponding to each first neural network layer remain unchanged, which is not only beneficial to improving the stability of the parameters adopted by each first neural network layer, but also beneficial to reducing the difficulty of the first machine learning model in the training process.
[0018] In one possible implementation, the execution device processes the first data through the first machine learning model, including: the execution device fuses the first feature information of the first data with the second feature information of the first data to obtain the updated first feature information, and the updated first feature information is used to obtain the first prediction result, and the first prediction result is one of the at least one prediction result corresponding to the first task. Among them, the first feature information corresponds to the first task, the second feature information corresponds to the second task, the first task is any one of the at least one task, and the second task is a task other than the first task in the at least one task. In this implementation, the second feature information corresponding to other tasks (that is, the second task) is integrated into the first feature information corresponding to the first task to obtain the updated first feature information, which is beneficial for the feature processing network corresponding to the first task to obtain richer information, and thus is beneficial for improving the accuracy of the prediction result output by the first machine learning model.
[0019] In one possible implementation, an execution device fuses first feature information of first data with second feature information of the first data to obtain updated first feature information, including: the execution device fuses the first feature information with the second feature information based on an attention mechanism to obtain the updated first feature information. In this implementation, not only is the second feature information fused with the first feature information to carry richer information in the updated first feature information, but the aforementioned fusion process based on the attention mechanism is also beneficial in that the updated first feature information pays more attention to information requiring attention, thereby improving the accuracy of the prediction results output by the first machine learning model.
[0020] In one possible implementation, the execution device fuses the first feature information with the second feature information based on the attention mechanism to obtain the updated first feature information, including: the execution device generates a first query feature, a first key feature, and a first value feature based on the first feature information corresponding to the first task, and generates a first attention matrix corresponding to the first task based on the first query feature and the first key feature; the execution device obtains a second attention matrix corresponding to the second task, and the second attention matrix is obtained based on the second feature information corresponding to the second task. The execution device fuses the first attention matrix and the second attention matrix to obtain a fusion result; based on the fusion result and the first value feature, the updated first feature information is generated. Exemplarily, the execution device can multiply the first fusion result and the first value feature to obtain the updated first feature information.
[0021] In this implementation method, a first attention matrix and a first value feature are obtained based on the first feature information, and a second attention matrix is obtained based on the second feature information. The first attention matrix and the second attention matrix are fused to obtain a fusion result. Then, based on the fusion result and the first value feature, an updated first feature information is generated (that is, the fusion of the second feature information into the first feature information is achieved). Through the aforementioned method, an implementation scheme for fusing the second feature information into the first feature information based on the attention mechanism is provided, which is simple and easy to operate. Moreover, the aforementioned fusion method is compatible with the process of updating the first feature information based on the attention mechanism, which further reduces the difficulty of implementation.
[0022] In one possible implementation, the first data is an image, and the at least one task includes any one or more of the following: image classification, object detection in an image, semantic segmentation in an image, segmenting an attention target from an image, text recognition in an image, image instance segmentation, pose estimation of a human figure in an image, or action recognition of a human figure in an image. This implementation provides the possibility of at least one different category of image processing task when the first data is an image, thereby improving the integration of this solution with actual application scenarios and increasing its flexibility.
[0023] In a second aspect, an embodiment of the present application provides a model training method that can be used in a multi-task processing scenario in the field of artificial intelligence. The method may include: a training device obtains first data and first information, where the first information indicates at least one task to be performed on the first data; the first data and the first information are input into a first machine learning model, and the first data is processed by the first machine learning model to obtain at least one prediction result output by the first machine learning model that corresponds one-to-one to at least one task. The training device trains the first machine learning model based on at least one correct result and at least one prediction result that correspond one-to-one to at least one task, as well as a loss function, where the loss function indicates the similarity between the prediction result and the correct result corresponding to each task in the at least one task.
[0024] In one possible implementation, the first machine learning model includes multiple neural network layers, and there is at least one first neural network layer among the multiple neural network layers. The training device processes the first data through the first machine learning model, including: the training device determines the parameters adopted by the first neural network layer based on the characteristic information of the first information; according to the parameters adopted by the first neural network layer, the first neural network layer is used to process the second data to obtain the processing result of the second data generated by the first neural network layer, where the second data is the first data or the characteristic information of the first data.
[0025] In one possible implementation, the training device processes the first data through a first machine learning model, including: the training device fuses the first feature information of the first data with the second feature information of the first data to obtain updated first feature information, and the updated first feature information is used to obtain a first prediction result, the first prediction result being one of at least one prediction result corresponding to the first task; wherein the first feature information corresponds to the first task, the second feature information corresponds to the second task, the first task is any one of the at least one task, and the second task is a task other than the first task of the at least one task.
[0026] In the second aspect of this application, the training device can also be used to execute the steps performed by the device in the first aspect and various possible implementation methods of the first aspect. The specific implementation methods, meanings of terms and beneficial effects brought about by the steps in each possible implementation method of the second aspect can all be referred to the first aspect and will not be repeated here.
[0027] On the third aspect, an embodiment of the present application provides a data processing device that can be used in a multi-task processing scenario in the field of artificial intelligence. The data processing device may include: an acquisition module for acquiring first data and first information, where the first information indicates at least one task to be performed on the first data; a processing module for inputting the first data and the first information into a first machine learning model, processing the first data through the first machine learning model, and obtaining at least one prediction result output by the first machine learning model that corresponds one-to-one to at least one task.
[0028] In the third aspect of the present application, the data processing device can also be used to execute the steps performed by the execution device in the first aspect and various possible implementation methods of the first aspect. The specific implementation methods, meanings of terms and beneficial effects brought about by the steps in various possible implementation methods of the third aspect can all be referred to the first aspect and will not be repeated here.
[0029] In a fourth aspect, an embodiment of the present application provides a model training device that can be used in a multi-task processing scenario in the field of artificial intelligence. The model training device may include: an acquisition module for acquiring first data and first information, where the first information indicates at least one task performed on the first data; a processing module for inputting the first data and the first information into a first machine learning model, processing the first data through the first machine learning model, and obtaining at least one prediction result output by the first machine learning model that corresponds one-to-one to at least one task; a training module for training the first machine learning model based on at least one correct result and at least one prediction result corresponding one-to-one to at least one task, and a loss function, where the loss function indicates the similarity between the prediction result and the correct result corresponding to each task in at least one task.
[0030] In the fourth aspect of this application, the model training device can also be used to execute the steps performed by the execution device in the second aspect and various possible implementation methods of the second aspect. The specific implementation methods, meanings of terms and beneficial effects brought about by the steps in each possible implementation method of the fourth aspect can all be referred to the second aspect and will not be repeated here.
[0031] In the fifth aspect, an embodiment of the present application provides an execution device, including a processor and a memory, the processor is coupled to the memory, the memory is used to store programs; the processor is used to execute the programs in the memory, so that the execution device executes the data processing method of the first aspect mentioned above.
[0032] In the sixth aspect, an embodiment of the present application provides a training device, including a processor and a memory, where the processor is coupled to the memory, the memory is used to store programs; and the processor is used to execute the programs in the memory, so that the training device executes the training method of the model of the second aspect mentioned above.
[0033] In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the method described in the first or second aspect above.
[0034] In an eighth aspect, an embodiment of the present application provides a computer program product, which includes a program. When the program runs on a computer, it enables the computer to execute the method described in the first or second aspect above.
[0035] In a ninth aspect, the present application provides a chip system, which includes a processor for supporting the implementation of the functions involved in the above-mentioned various aspects, such as sending or processing the data and / or information involved in the above-mentioned methods. In one possible design, the chip system also includes a memory, which is used to store program instructions and data necessary for the terminal device or communication device. The chip system can be composed of a chip, or it can include a chip and other discrete devices. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] FIG1 is a schematic diagram of the structure of an artificial intelligence main framework provided in an embodiment of the present application;
[0037] FIG2 is a system architecture diagram of a data processing system provided in an embodiment of the present application;
[0038] FIG3 is a flow chart of a data processing method provided in an embodiment of the present application;
[0039] FIG4 is another flow chart of a data processing method according to an embodiment of the present application;
[0040] FIG5 is a schematic diagram of extracting features from second data using any first neural network layer according to an embodiment of the present application;
[0041] FIG6 is a schematic diagram of an operation of performing a first feature extraction phase by a first machine learning model according to an embodiment of the present application;
[0042] FIG7 is a schematic diagram of obtaining a first attention matrix and a first value feature based on first feature information provided by an embodiment of the present application;
[0043] FIG8 is a schematic diagram of an operation of performing a second feature extraction phase by a first machine learning model according to an embodiment of the present application;
[0044] FIG9 is a schematic diagram of a first machine learning model provided in an embodiment of the present application;
[0045] FIG10 is a schematic diagram of a residual network provided in an embodiment of the present application;
[0046] FIG11 is a schematic diagram of a structure of each residual block provided in an embodiment of the present application;
[0047] FIG12 is a schematic diagram of a neural network used when performing the first feature extraction phase through the feature extraction network of the first machine learning model provided in an embodiment of the present application;
[0048] FIG13 is a flow chart of a method for training a model according to an embodiment of the present application;
[0049] FIG14 is a schematic structural diagram of a data processing device provided in an embodiment of the present application;
[0050] FIG15 is a schematic structural diagram of a model training device provided in an embodiment of the present application;
[0051] FIG16 is a schematic diagram of another structure of an execution device provided in an embodiment of the present application;
[0052] FIG17 is a schematic diagram of another structure of the training device provided in an embodiment of the present application;
[0053] FIG18 is a schematic structural diagram of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION
[0054] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
[0055] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.
[0056] First, let's describe the overall workflow of an AI system. See Figure 1, which shows a schematic diagram of the AI framework. This framework will be explained from two perspectives: the "intelligent information chain" (horizontal axis) and the "IT value chain" (vertical axis). The "intelligent information chain" reflects the entire process from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. Throughout this process, data undergoes a condensed journey from "data-information-knowledge-wisdom." The "IT value chain," spanning the underlying infrastructure of human intelligence, information (provided and processed by technology), and the system's industrial ecosystem, reflects the value that AI brings to the information technology industry.
[0057] (1) Infrastructure
[0058] The infrastructure provides computing power for AI systems, enabling communication with the outside world and providing support through a basic platform. Communication with the outside world is achieved through sensors; computing power is provided by smart chips, which can specifically adopt hardware acceleration chips such as central processing units (CPUs), embedded neural network processing units (NPUs), graphics processing units (GPUs), application-specific integrated circuits (ASICs), or field programmable gate arrays (FPGAs); the basic platform includes related platform guarantees and support such as distributed computing frameworks and networks, and can include cloud storage and computing, and interconnected networks. For example, sensors communicate with the outside world to obtain data, which is then provided to the smart chips in the distributed computing system provided by the basic platform for calculation.
[0059] (2) Data
[0060] Data above the infrastructure layer represents data sources for AI. This data includes graphics, images, voice, and text, as well as IoT data from traditional devices. This includes business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.
[0061] (3) Data processing
[0062] Data processing generally includes data training, machine learning, deep learning, search, reasoning, decision-making, etc.
[0063] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.
[0064] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.
[0065] Decision-making refers to the process of making decisions after intelligent information is reasoned, and usually provides functions such as classification, sorting, and prediction.
[0066] (4) General ability
[0067] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.
[0068] (5) Smart products and industry applications
[0069] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical application. Its application areas mainly include: smart terminals, smart manufacturing, smart transportation, smart homes, smart medical care, smart security, autonomous driving, smart cities, etc.
[0070] The method provided in this application can be applied to various application fields of artificial intelligence, and optionally, can be used in application scenarios where one or more tasks may need to be completed simultaneously on the same data. The aforementioned one or more tasks may all be image processing tasks; or the aforementioned one or more tasks may all be visual perception tasks; or the aforementioned one or more tasks may all be natural language processing (NLP) tasks related to speech and semantics, etc. The following provides examples of multiple application scenarios of this application.
[0071] Application Scenario 1: Image Processing Task
[0072] For example, in the field of smart terminals, when a user takes photos or videos with a mobile phone, the phone can perform one or more image classification tasks and target detection tasks on the images of the aforementioned shooting scene captured by the camera. The aforementioned image classification task is performed to identify the category of the aforementioned shooting scene. The "category of the aforementioned shooting scene" may include night scene shooting, landscape shooting, food shooting, or other categories, etc. The aforementioned target detection task is performed to determine the category and location of objects in the aforementioned shooting scene. For example, the categories of objects in the aforementioned shooting scene may include people, animals, flowers, plants, or other categories, etc. The mobile phone can automatically determine the shooting mode suitable for the current shooting scene based on the category of the shooting scene and the categories of the objects in the shooting scene.
[0073] For another example, in the field of smart terminals, when a user conducts an online meeting through a computer, he or she may extract the content of a certain video frame in the online meeting. The computer can then perform target detection on the aforementioned video frame to determine the text area and image area included in the video frame. Then, based on the result of the target detection, the computer can simultaneously perform the following tasks on the video frame: perform semantic segmentation on the video frame to extract the image from the video frame in a timely manner; perform text recognition on the text area of the video frame to extract the text from the video frame in a timely manner; and perform table recognition on the text area of the video frame to extract the table from the video frame in a timely manner.
[0074] For example, in the field of smart terminals, when a user follows the movements shown on a smart display screen, the processor of the smart display screen can capture the user's image through a camera, perform user recognition tasks and human motion recognition tasks based on the user's image, and provide timely feedback to the user when the user's movements are wrong, etc.
[0075] Application Scenario 2: Natural Language Processing Tasks
[0076] Natural language processing (NLP) is the process of systematically analyzing, understanding, and extracting information from text data using machine learning models. In applications such as smart devices, smart homes, and autonomous driving, machine learning models can be used to perform multiple NLP tasks simultaneously on the same data.
[0077] In the various application areas mentioned above, by using the aforementioned machine learning models, we can manage very large amounts of text data, perform a large number of automated tasks, and solve a variety of problems, such as automatic summarization, machine translation (MT), named entity recognition (NER), relation extraction (RE), information extraction (IE), sentiment analysis, speech recognition, question answering, and topic segmentation.
[0078] For example, natural language processing tasks can be divided into the following categories.
[0079] Sequence labeling: Each word in a text requires the machine learning model to assign a classification category based on the context, such as Chinese word segmentation, part-of-speech tagging, named entity recognition, or semantic role labeling.
[0080] Classification tasks: The machine learning model outputs a classification value for the entire input text, such as sentiment classification, topic classification, or whether the grammar is correct.
[0081] Sentence relationship inference: A machine learning model takes two texts as input and uses them to determine whether the two texts have a certain nominal relationship. Examples include question-answering systems, semantic rewriting, and natural language inference.
[0082] Generative tasks: Input a piece of text and generate another piece of text through a machine learning model. Examples include machine translation, automatic summarization, or writing poetry and sentences.
[0083] Information extraction task: obtain at least one type of information from the input text through a machine learning model.
[0084] For example, when people from different countries hold a meeting, they may need to translate a text into multiple languages. In this case, multiple machine translation tasks need to be performed on the text simultaneously, and each of the multiple machine translation tasks is used to translate the text into one language, and so on.
[0085] It should be noted that the method provided in this application can also be applied to other scenarios. The above examples of various application scenarios of this application are only for the convenience of understanding this solution and are not used to limit this solution.
[0086] Since in many scenarios, there is a need to perform N tasks on the same first data, N is an integer greater than or equal to 2; the first machine learning model deployed on the device can be a machine learning model that can perform N tasks simultaneously, but since the execution frequency of the N tasks may be different, each time the first machine learning model is called, it is not necessarily necessary to perform all of the N tasks on the first data input into the first machine learning model.
[0087] In order to avoid wasting computer resources, the present application provides a data processing method. Before introducing the method provided by the present application in detail, please refer to Figure 2. Figure 2 is a system architecture diagram of the data processing system provided by an embodiment of the present application. In Figure 2, the data processing system 200 includes a training device 210, a database 220, an execution device 230 and a data storage system 240, and the execution device 230 includes a computing module 231.
[0088] During the training phase of the first machine learning model 201, a training data set is stored in the database 220, the training device 210 generates the first machine learning model 201, and iteratively trains the first machine learning model 201 using the training data set to obtain the trained first machine learning model 201. The first machine learning model 201 can be specifically expressed as a neural network or a non-neural network model. In the embodiments of the present application, only the first machine learning model 201 expressed as a neural network is used as an example for description.
[0089] The first and second convolutional neural networks obtained by the training device 210 can be applied to different systems or devices, such as mobile phones, tablets, laptops, virtual reality (VR) devices, monitoring systems, radar data processing systems, etc. The execution device 230 can call data, code, etc. from the data storage system 240, or store data, instructions, etc. in the data storage system 240. The data storage system 240 can be located within the execution device 230, or the data storage system 240 can be an external memory relative to the execution device 230.
[0090] During the application phase of the first machine learning model 201, after determining at least one task to be performed on the first data, the execution device 230 may use the first machine learning model 201 to generate at least one prediction result corresponding to the at least one task. Specifically, please refer to FIG3 , which is a schematic diagram of a data processing method provided in an embodiment of the present application. In 301 , the execution device 230 obtains first data and first information, where the first information indicates at least one task to be performed on the first data.
[0091] 302. The execution device 230 inputs the first data and the first information into the first machine learning model 201, processes the first data through the first machine learning model 201, and obtains at least one prediction result output by the first machine learning model 201 corresponding to at least one task.
[0092] In an embodiment of the present application, first information is added to the input of the first machine learning model 201, and the first information is used to indicate that at least one task needs to be performed on the first data. The first machine learning model 201 outputs at least one prediction result corresponding one-to-one to the at least one task. That is, the first machine learning model 201 can know which tasks need to be performed on the first data based on the first information, and then can adaptively perform the required tasks on the first data, avoiding the generation of redundant prediction results, thereby avoiding the waste of computer resources.
[0093] In some embodiments of the present application, referring to FIG. 2 , the execution device 230 and the client device may be integrated into the same device, and the user may interact directly with the execution device 230. For example, when the client device is a mobile phone or a tablet, the execution device 230 may be a module in the host processor (Host CPU) of the mobile phone or tablet that performs data processing using the first machine learning model. The execution device 230 may also be a graphics processing unit (GPU) or a neural network processor (NPU) in the mobile phone or tablet. The GPU or NPU is mounted on the host processor as a coprocessor, and the host processor assigns tasks.
[0094] It is worth noting that FIG2 is only a schematic diagram of the architecture of two data processing systems provided by an embodiment of the present invention, and the positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in other embodiments of the present application, the execution device 230 and the client device can be independent devices, and the execution device 230 is configured with an input / output (I / O) interface to interact with the client device. After the client device determines the first data and the first information, the client device sends the first data and the first information to the execution device 230 through the I / O interface. After the execution device 230 generates at least one prediction result corresponding to at least one task through the first machine learning model / rule 201 in the computing module 231, the aforementioned prediction result can be returned to the client device through the I / O interface and provided to the user.
[0095] In combination with the above description, the following describes the specific implementation process of the training phase and application phase of the method provided in the embodiment of the present application.
[0096] 1. Application stage
[0097] In the embodiment of the present application, the application stage describes a process in which the execution device 230 processes the first data using the first machine learning model 201 that has performed the training operation. Specifically, please refer to FIG4 , which is another flow chart of the data processing method provided in the embodiment of the present application. The data processing method provided in the embodiment of the present application may include:
[0098] 401. Acquire first data and first information, where the first information indicates at least one task performed on the first data.
[0099] In an embodiment of the present application, the first machine learning model deployed on the execution device has the ability to simultaneously perform N tasks on the input first data. After determining which of the N tasks need to be performed on the first data, the first data and first information can be obtained. The first information is used to indicate that M of the N tasks need to be performed on the first data, where N is an integer greater than 1 and M is an integer greater than or equal to 1.
[0100] Exemplarily, if the first data is an image, the above-mentioned "N tasks" and "M tasks out of N tasks" may all be image processing tasks, and the "N tasks" include N different image processing tasks. Exemplarily, the categories of the N different image processing tasks may include any one or more of the following: image classification, target detection on images, semantic segmentation on images, segmentation of attention targets from images, text recognition on images, image instance segmentation, posture estimation of human bodies in images, action recognition of human bodies in images, or other tasks performed on images, etc. The specific tasks included in the N tasks need to be flexibly determined in combination with the actual application scenarios, and are not limited in the embodiments of the present application. The possibility of N different categories of image processing tasks is provided when the first data is an image, which improves the degree of integration of the present solution with the actual application scenarios and also improves the flexibility of the present solution.
[0101] If the first data is text, then the N tasks and the M tasks can all be text-related natural language processing tasks, and the N tasks can include N different text-related natural language processing tasks. If the first data is audio, then the N tasks and the M tasks can all be audio processing tasks, and the N tasks can include N different text-related audio processing tasks. When the first data is in other forms, the N tasks and the M tasks can also all be tasks for processing the first data in other forms, etc. This is not exhaustive in the embodiments of this application.
[0102] Optionally, the first information can be specifically expressed as a first vector, which can include N elements corresponding to N tasks. When the value of any one of the N elements (hereinafter referred to as a "target element" for ease of description) is the first value, it means that a task corresponding to the target element needs to be executed on the first data. When the value of the target element is the second value, it means that the task corresponding to the target element does not need to be executed on the first data. The first value and the second value are different. For example, the first value can be 1 and the second value can be 0; or the first value can be 0 and the second value can be 0; or the first value can be 1 and the second value can be 2, and so on. It should be noted that the examples here are only for the convenience of understanding this solution and are not used to limit this solution.
[0103] Exemplarily, the first data is an image, the value of N is 4, and the four tasks performed on the first data include: target detection on the image, semantic segmentation on the image, text recognition on the image, and segmenting the attention target from the image. When the first information is (1, 0, 1, 0), it can indicate that the tasks that need to be performed on the input first data include: target detection on the image and text recognition on the image; when the first information is (1, 1, 0, 0), it can indicate that the tasks that need to be performed on the input first data include: target detection on the image and semantic segmentation on the image, etc. It should be understood that the example here is only for the convenience of understanding the concept of "the first information indicates at least one task to be performed on the first data" and is not used to limit this solution.
[0104] 402. Input the first data and the first information into a first machine learning model, perform feature extraction on the first data through the first machine learning model, and obtain at least one third feature information of the first data.
[0105] In an embodiment of the present application, after obtaining the first data and the first information, the execution device can input the first data and the first information into a first machine learning model, perform feature extraction on the first data through the first machine learning model, and obtain M third feature information of the first data corresponding to M tasks one by one. It should be noted that the concepts of "first feature information" and "second feature information" will be described later.
[0106] In which, the feature extraction network in the first machine learning model may include multiple neural network layers. Optionally, the aforementioned multiple neural network layers may include one or more first neural network layers. The parameters of each first neural network layer are determined based on the first information, that is, the first information is used to determine the parameters adopted by each first neural network layer; when the input first information is different, the parameters adopted by the first neural network layer may be different.
[0107] Any first neural network layer included in the feature extraction network in the first machine learning model can be a convolution layer, a fully connected layer, a neural network layer for linear changes, or other types of neural network layers, etc. The specific configuration can be flexibly determined based on the actual application scenario and is not limited in the embodiments of the present application.
[0108] For any one of the at least one first neural network layers (hereinafter referred to as the "target neural network layer" for the convenience of description), step 402 may include: the execution device determines the parameters adopted by the target neural network layer based on the first information; since the type of operation performed by the first neural network layer is preset, after the execution device determines the parameters adopted by the target neural network layer, it can use the first neural network layer to perform feature extraction on the second data according to the parameters adopted by the first neural network layer to obtain the processing result of the second data generated by the first neural network layer, where the second data is the first data or feature information of the first data.
[0109] Among them, if the target neural network layer is the first neural network layer used by the first machine learning model to process the first data, the second data can be the first data, and the processing result of the second data is the feature information of the first data; if the target neural network layer is not the first neural network layer used by the first machine learning model to process the first data, the second data can be the feature information of the first data, and the processing result of the second data can be the updated feature information of the first data.
[0110] Both “feature information of the first data” and “updated feature information of the first data” can be understood as feature information of the first data, that is, the processing result of the second data may include feature maps of multiple channels of the first data; optionally, when the first information is different, the number of feature maps in the processing result of the second data generated by the target neural network layer may be different.
[0111] Exemplarily, if the target neural network layer is a convolutional layer, when the first input information is different, the size of the convolution kernel adopted by the target neural network layer can be different, thereby achieving different parameters adopted by the target neural network layer; or, if the target neural network layer is a convolutional layer, when the first input information is different, the number of convolution kernels adopted by the target neural network layer can be different, thereby achieving different parameters adopted by the target neural network layer, and so on. It should be noted that when the target neural network layer is expressed as other types of neural network layers, "the parameters adopted by the first neural network layer are different" can also be expressed in other forms. The examples here are only to prove the feasibility of this solution and are not used to limit this solution.
[0112] The execution device can execute "determining the parameters adopted by the target neural network layer based on the first information" in a variety of ways. In one case, the execution device extracts features of the first information through the feature extraction network of the first machine learning model to obtain feature information of the first information; based on the feature information of the first information, the parameters adopted by the target neural network layer are determined; when the input first information is different, the parameters adopted by the target neural network layer may be different.
[0113] Exemplarily, the execution device may implement “determining the parameters used by the target neural network layer based on the characteristic information of the first information” in a variety of ways. In one implementation, the execution device may obtain first parameters corresponding to the target neural network layer; determine second parameters corresponding to the target neural network layer based on the characteristic information of the first information; determine the parameters used by the target neural network layer based on the first and second parameters corresponding to the target neural network layer; and determine a set of parameters used by each first neural network layer using the aforementioned methods.
[0114] For example, the "first parameter corresponding to the target neural network layer" can be understood as a parameter that is independent of the task. That is, the "first parameter corresponding to the target neural network layer" is used as a set of bases. Regardless of which tasks among the N tasks are included in the M tasks, the aforementioned "first parameter corresponding to the target neural network layer" will be obtained. The "second parameter corresponding to the target neural network layer" can be understood as a parameter related to the task. When the first information is different (that is, when the M tasks executed on the first data are different), the second parameter corresponding to the target neural network layer can be different.
[0115] Exemplarily, the first parameter and the second parameter corresponding to the target neural network layer can both be expressed as matrices.
[0116] Exemplarily, the feature extraction network of the first machine learning model may include a first module corresponding to the target neural network layer, and the execution device may generate a second parameter corresponding to the target neural network layer through the aforementioned first module based on the feature information of the first information. The feature extraction network of the first machine learning model may include a first module corresponding to at least one first neural network layer, that is, each first neural network layer has a first module corresponding thereto. Alternatively, multiple first neural network layers in the feature extraction network of the first machine learning model may also share a first module; for example, if the feature extraction network of the first machine learning model includes multiple residual blocks, a residual block may include multiple convolutional layers, and the convolutional layers in the same residual block may share a first module, etc. It should be noted that the relationship between the "first neural network layer" and the "first module" can be set based on the actual application scenario. The example here is only for the convenience of understanding this solution and is not used to limit this solution.
[0117] The execution device may linearly weight the first parameter and the second parameter corresponding to the target neural network layer to obtain the parameters adopted by the target neural network layer; or, the execution device may perform point multiplication or addition on the first parameter and the second parameter corresponding to the target neural network layer to obtain the parameters adopted by the target neural network layer; or, the execution device may also perform other calculation operations on the first parameter and the second parameter corresponding to the target neural network layer to obtain the parameters adopted by the target neural network layer. The specific operation can be flexibly determined in combination with the actual application scenario and is not limited in the embodiments of the present application.
[0118] For a more intuitive understanding of this solution, please refer to Figure 5, which is a schematic diagram of an embodiment of the present application for extracting features from the second data using any first neural network layer. In Figure 5, taking the value of N as 3 as an example, the N tasks include Task 1 (Task 1), Task 2 (Task 2), and Task 3 (Task 3), and the first information indicates that Task 1 of the aforementioned three tasks needs to be executed on the input first data. Feature extraction is performed on the first information through the feature extraction network of the first machine learning model to obtain feature information of the first data; the execution device can generate a second parameter corresponding to the target neural network through the first module corresponding to the target neural network layer (that is, any first neural network layer in the first machine learning model) in the first machine learning model based on the feature information of the first information.
[0119] The execution device can generate the parameters used by the target neural network layer based on the second parameters and the first parameters corresponding to the target neural network layer. In Figure 5, the parameters used by the target neural network layer are taken as an example of a 4x4 matrix. The aforementioned 4x4 matrix is divided into the parameters of four 1x4 convolution kernels. After determining the parameters used by the target neural network layer, the execution device processes the second data through the target neural network layer to obtain the processing results of the second data generated by the target neural network layer. It should be understood that the example in Figure 5 is only for the convenience of understanding this solution and is not used to limit this solution.
[0120] In an embodiment of the present application, the parameters adopted by each first neural network layer are decoupled into task-independent parameters (i.e., first parameters) and task-related parameters (i.e., second parameters) corresponding to the first neural network layer. After determining which tasks need to be performed on the input first data, the task-independent parameters corresponding to the first neural network layer can be determined based on the characteristic information of the first information, and then the parameters finally adopted by each first neural network layer are determined based on the task-independent parameters and the task-related parameters. Since the first information will affect the second parameters corresponding to each first neural network layer, it is beneficial to achieve adaptability between the parameters adopted by each first neural network layer and the first information. In addition, no matter which tasks need to be performed on the input first data, the first parameters corresponding to each first neural network layer remain unchanged, which is not only beneficial to improving the stability of the parameters adopted by each first neural network layer, but also beneficial to reducing the difficulty of the first machine learning model in the training process.
[0121] In another implementation, the execution device may directly determine the parameters adopted by the target neural network layer based on the feature information of the first information; the execution device may determine a set of parameters adopted by each first neural network layer in the aforementioned manner. Exemplarily, the feature extraction network of the first machine learning model may include a second module corresponding to the target neural network layer, and the execution device may generate the parameters adopted by the target neural network layer through the aforementioned second module based on the feature information of the first information. Among them, the feature extraction network of the first machine learning model may include a second module corresponding to at least one first neural network layer, that is, each first neural network layer has a second module corresponding thereto. Alternatively, multiple first neural network layers in the feature extraction network of the first machine learning model may also share a first module, etc., which is not limited in the embodiments of the present application.
[0122] In another case, the execution device may be pre-configured with multiple sets of parameters that can be used by each first neural network layer, and the aforementioned multiple sets of parameters correspond to the multiple combinations corresponding to the aforementioned N tasks; the aforementioned multiple sets of parameters include a set of parameters corresponding to each of the multiple combinations corresponding to the aforementioned N tasks. For example, the value of N is 5. When the first information is expressed as (0, 0, 1, 1, 1), it represents a combination of 5 tasks. The aforementioned combination means that the third, fourth, and fifth tasks need to be executed on the first input data. When the first information is expressed as (0, 1, 0, 0, 1), it represents another combination of 5 tasks. The aforementioned combination means that the second and fifth tasks need to be executed on the first input data, and so on. It should be understood that the examples given here are only for the convenience of understanding the concept of "multiple combinations corresponding to N tasks" and are not used to limit this solution.
[0123] Step 402 may include: the execution device may obtain a set of parameters corresponding to the first information from multiple sets of parameters that the target neural network layer can adopt, that is, determining a set of parameters actually adopted by the target neural network layer; the execution device may use the aforementioned method to determine a set of parameters adopted by each first neural network layer.
[0124] In an embodiment of the present application, when the number of parameters adopted by the first machine learning model is larger, more computer resources are consumed when using the first machine learning model for data processing, and more abundant information can be mined from the input first data; correspondingly, when the number of parameters adopted by the first machine learning model is smaller, fewer computer resources are consumed when using the first machine learning model for data processing, and the information mined from the input first data will be relatively less; the first machine learning model has the ability to execute N tasks simultaneously. When all of the N tasks are executed by the first machine learning model, each first neural network layer can adopt more parameters, but not all of the N tasks are executed by the first machine learning model each time (that is, the M tasks that need to be executed on the first data may be part of the N tasks). Determining the parameters adopted by each first neural network layer based on the first information is conducive to achieving adaptation between the parameters adopted by each first neural network layer and the "M tasks that need to be executed on the first data" to avoid waste of computer resources.
[0125] In addition, since the feature extraction network in the first machine learning model consumes a lot of computer resources in the process of processing the first data using the first machine learning model, adjusting the parameters used by the neural network layer in the feature extraction network based on the first information is conducive to significantly reducing the waste of computer resources.
[0126] Optionally, the process of the execution device "extracting features of the first data through the first machine learning model" may include a first feature extraction stage and a second feature extraction stage. The feature information of the first data obtained in the first feature extraction stage is feature information shared by M tasks, and the second feature extraction stage is used to obtain M third feature information of the first data corresponding one-to-one to the M tasks based on the aforementioned shared feature information.
[0127] Exemplarily, all first neural network layers included in the feature extraction network of the first machine learning model may be neural network layers that perform the first feature extraction stage; or, multiple first neural network layers included in the feature extraction network of the first machine learning model may exist in both the first feature extraction stage and the second feature extraction stage; or, all first neural network layers included in the feature extraction network of the first machine learning model may be neural network layers that perform the second feature extraction stage.
[0128] In order to understand this solution more intuitively, please refer to Figure 6, which is a schematic diagram of the operation of performing the first feature extraction stage through the first machine learning model provided in an embodiment of the present application. As shown in Figure 6, a plurality of convolution modules are used in the process of performing the operation of the first feature extraction stage on the input image (that is, an example of the first data) through the feature extraction network of the first machine learning model. In Figure 6, the convolution module 2 is described as an example. The convolution module 2 includes two first neural network layers, namely the first neural network layer 1 and the first neural network layer 2. The first neural network layer 1 and the first neural network layer 2 are both convolution layers for performing convolution operations. The input of the convolution module 2 is the feature information of the aforementioned image.
[0129] Before using the first neural network layer 1 to perform a convolution operation, the execution device first generates a second parameter corresponding to the first neural network layer 1 through the first module in the first machine learning model based on the feature information of the first information, and then determines the parameters adopted by the first neural network layer 1 based on the first parameters and second parameters corresponding to the first neural network layer 1, and then performs a convolution operation on the second data (that is, the feature information of the image input into the convolution module 2) through the first neural network layer 1 to obtain updated feature information of the image generated by the first neural network layer 1.
[0130] Correspondingly, before performing the convolution operation using the first neural network layer 2, the second parameter corresponding to the first neural network layer 2 is first obtained. In Figure 6, the first neural network layer 1 and the second neural network layer 2 share the same first module as an example, that is, the second parameters corresponding to the first neural network layer 1 and the first neural network layer 2 are the same. The execution device determines the parameters used by the first neural network layer 2 based on the first and second parameters corresponding to the first neural network layer 2, and then performs a convolution operation on the second data (that is, the updated feature information of the image generated by the first neural network layer 1) through the first neural network layer 2 to obtain the updated feature information of the image generated by the first neural network layer 2.
[0131] The feature extraction network of the first machine learning model uses multiple convolution modules to perform the first feature extraction stage operation on the input image. The process of other convolution modules performing convolution operations can refer to the description of convolution module 2. The specific implementation methods of other convolution modules will not be described in detail here. It should be understood that the example in Figure 6 is only for the convenience of understanding this solution and is not used to limit this solution.
[0132] For example, after obtaining feature information shared by M tasks through the feature extraction network of the first machine learning model, the execution device can update the shared feature information through M adapters corresponding one-to-one to the M tasks, thereby obtaining a third feature information generated by each of the M adapters, that is, obtaining M third feature information of the first data corresponding one-to-one to the M tasks. Each of the aforementioned M adapters can include one or more neural network layers, and the specific design of each adapter can be flexibly determined based on actual conditions and is not limited here.
[0133] Alternatively, after the execution device enters the second feature extraction stage in the process of processing the first data through the first machine learning model, the processing process of any one of the M tasks (hereinafter referred to as the "first task" for the convenience of description) may include: the execution device can obtain the first feature information of the first data corresponding to the first task based on the feature information of the first data obtained in the first feature extraction stage (that is, the shared feature information).
[0134] The execution device fuses the first feature information of the first data with the (M-1) second feature information of the first data to obtain updated first feature information. The updated first feature information corresponding to the first task is used to obtain a first prediction result, where the first prediction result is one of the at least one prediction results corresponding to the first task; the second feature information of the first data corresponds to a second task, where the first task is any one of the M tasks, and the second task is a task other than the first task among the M tasks to be executed on the first data.
[0135] The execution device can obtain the updated first feature information corresponding to each of the M tasks in the above manner, and the updated first feature information corresponding to each of the M tasks is determined as the third feature information corresponding to the task.
[0136] Optionally, the execution device may input the shared feature information of the first data obtained in the first feature extraction stage into the first adapter corresponding to the first task, and obtain the above-mentioned first feature information generated by the adapter corresponding to the first task. Correspondingly, the execution device may input the shared feature information of the first data obtained in the first feature extraction stage into the second adapter corresponding to each second task, and obtain the second feature information corresponding to each second task generated by the adapter corresponding to each second task. Exemplarily, the aforementioned "first adapter" and "second adapter" may each include multiple neural network layers.
[0137] The execution device can implement "fusing the first feature information of the first data with the (M-1) second feature information of the first data to obtain updated first feature information" in a variety of ways. In one implementation, the execution device can fuse the first feature information with the (M-1) second feature information based on an attention mechanism to obtain updated first feature information. This not only achieves the fusion of the second feature information with the first feature information to carry richer information in the updated first feature information, but also, executing the aforementioned fusion process based on the attention mechanism helps to make the updated first feature information pay more attention to the information that needs attention, which helps to improve the accuracy of the prediction results output by the first machine learning model.
[0138] For example, in one scenario, the execution device may generate a first query feature, a first key feature, and a first value feature based on first feature information corresponding to the first task, and generate a first attention matrix corresponding to the first task based on the first query feature and the first key feature. The execution device obtains a second attention matrix corresponding to the second task, the second attention matrix being obtained based on second feature information corresponding to the second task. The execution device fuses the first attention matrix and the second attention matrix to obtain a first fusion result; and generates updated first feature information based on the first fusion result and the first value feature.
[0139] The execution device may perform a first linear transformation operation on the first feature information through a neural network layer in a feature extraction network of the first machine learning model to obtain a first query feature; perform a second linear transformation operation on the first feature information through a neural network layer in the feature extraction network of the first machine learning model to obtain a first key feature; the execution device multiplies the first query feature and the first key feature to obtain a first attention matrix. The execution device may perform a third linear transformation operation on the first feature information through a neural network layer in the feature extraction network of the first machine learning model to obtain a first value feature.
[0140] Optionally, any one or more of the following neural network layers may be the first neural network layer: a neural network layer that performs a first linear transformation operation on the first feature information, a neural network layer that performs a second linear transformation operation on the first feature information, or a neural network layer that performs a third linear transformation operation on the first feature information. It should be noted that the specific implementation process of "determining parameters used by a first neural network layer based on the first information" can be found in the above description and will not be elaborated here.
[0141] Alternatively, the neural network layer that performs the first linear transformation operation on the first feature information, the neural network layer that performs the second linear transformation operation on the first feature information, and the neural network layer that performs the third linear transformation operation on the first feature information may not all be the first neural network layer, that is, the parameters of the aforementioned neural network layers may not all be determined based on the first information.
[0142] To understand this solution more intuitively, please refer to Figure 7, which is a schematic diagram of obtaining a first attention matrix and a first value feature based on the first feature information provided in an embodiment of the present application. In Figure 7, the neural network layer that performs the third linear transformation operation on the first feature information is the first neural network layer, and the neural network layer that performs the first linear transformation operation on the first feature information and the neural network layer that performs the second linear transformation operation on the first feature information are not the first neural network layer. As shown in Figure 7, after obtaining the shared feature information, the execution device inputs the shared feature information into the adapter 1 corresponding to task 1 to obtain the first feature information corresponding to task 1; performs a first linear transformation operation on the first feature information to obtain a first query feature; performs a second linear transformation operation on the first feature information to obtain a first key feature; and multiplies the first query feature and the first key feature to obtain a first attention matrix.
[0143] The execution device can generate a second parameter corresponding to the neural network layer for performing the third linear transformation operation through the first module based on the characteristic information of the first information. It should be noted that the first module in Figure 7 and the first module in Figure 7 can be different first modules; the execution device determines the parameters adopted by the neural network layer for performing the third linear transformation operation based on the first parameter and the second parameter corresponding to the neural network layer for performing the third linear transformation operation, and then performs the third linear transformation operation on the first characteristic information to obtain a first value feature. It should be understood that the example in Figure 7 is only for the convenience of understanding this solution and is not used to limit this solution.
[0144] The specific implementation method of the execution device "generating a second attention matrix corresponding to the second task based on the second feature information corresponding to the second task" is similar to the specific implementation method of "generating a first attention matrix corresponding to the first task based on the first feature information corresponding to the first task". The difference is that "first task" is replaced by "second task", "first feature information" is replaced by "second feature information", and "first attention matrix" is replaced by "second attention matrix". Please refer to the above description for details and will not be repeated here.
[0145] For example, the method used for the above-mentioned "fusion" can be addition, weighted summation, multiplication, or other fusion methods, etc., which can be determined in combination with actual application scenarios and are not exhaustive here. For example, the execution device can multiply the first fusion result and the first value feature to obtain updated first feature information.
[0146] In an embodiment of the present application, a first attention matrix and a first value feature are obtained based on the first feature information, a second attention matrix is obtained based on the second feature information, the first attention matrix and the second attention matrix are fused, and after obtaining the fusion result, an updated first feature information is generated based on the fusion result and the first value feature (that is, the fusion of the second feature information into the first feature information is realized); through the aforementioned method, an implementation scheme for fusing the second feature information into the first feature information based on the attention mechanism is provided, which is simple and easy to operate; and the aforementioned fusion method is compatible with the process of updating the first feature information based on the attention mechanism, further reducing the difficulty of implementation.
[0147] In another scenario, the execution device may generate a first query feature, a first key feature, and a first value feature based on the first feature information corresponding to the first task, and generate a first attention matrix corresponding to the first task based on the first query feature and the first key feature. The execution device obtains a second value feature corresponding to the second task, and the second value feature is obtained based on the second feature information corresponding to the second task. The execution device fuses the first value feature and the second value feature to obtain a second fusion result; and generates updated first feature information based on the first attention matrix and the second fusion result. For example, the execution device may multiply the first attention matrix and the second fusion result to obtain updated first feature information.
[0148] In another implementation, the execution device can determine the first weight of the first feature information, determine the second weight of each of the (M-1) second feature information, perform weighted summation on the first feature information and the (M-1) second feature information, and obtain the updated first feature information; wherein, the sum of the (M-1) second weights corresponding to the (M-1) second feature information is less than the first weight.
[0149] It should be noted that the execution device can also fuse the first feature information of the first data with the (M-1) second feature information of the first data in other ways, which is not limited in the embodiments of the present application.
[0150] For a more intuitive understanding of this solution, please refer to Figure 8, which is a schematic diagram of the operation of performing the second feature extraction stage through the first machine learning model provided in an embodiment of the present application. As shown in Figure 8, the first information indicates that tasks 1 and 2 need to be performed on the first data input into the first machine learning model, and task 3 does not need to be performed on the aforementioned first data. The path shown by the solid line in Figure 8 transmits valid data, and the path shown by the dotted line can transmit empty data. After obtaining the shared feature information of the first data through the first feature extraction stage, during the process of performing the operations of the second feature extraction stage through the feature extraction network of the first machine learning model, the execution device can input the shared feature information of the first data into adapter 1 corresponding to task 1 to obtain the first feature information corresponding to task 1 (i.e., the input generated by adapter 1 in Figure 8). The Q Linear in Figure 8 is used to perform a first linear transformation operation on the first feature information corresponding to task 1 to obtain the query feature 1 corresponding to task 1; the K Linear in Figure 8 is used to perform a second linear transformation operation on the first feature information corresponding to task 1 to obtain the key feature 1 corresponding to task 1; and the V Linear in Figure 8 is used to perform a third linear transformation operation on the first feature information corresponding to task 1 to obtain the value feature 1 corresponding to task 1. The neural network layer used to perform the third linear transformation operation on the first feature information corresponding to task 1 is the first neural network layer, that is, the parameters used by the neural network layer used to perform the third linear transformation operation on the first feature information corresponding to task 1 are obtained based on the first information.
[0151] The execution device generates an attention matrix 1 corresponding to task 1 and a value feature corresponding to task 1 through the linear conversion module 1. The specific implementation method of all the steps performed by the execution device through the linear conversion module 1 can be found in the above description of Figure 7, which will not be repeated here.
[0152] Similarly, the execution device generates an attention matrix 2 corresponding to task 2 and a value feature 2 corresponding to task 2 through the linear transformation module 1 based on the attention mechanism. The specific implementation method of all steps performed by the execution device through the linear transformation module 2 based on the attention mechanism is similar to the specific implementation method of all steps performed by the linear transformation module 1, and will not be repeated here.
[0153] After obtaining attention matrix 1, value feature 1, attention matrix 2, and value feature 2, the execution device fuses attention matrix 1 and attention matrix 2 to obtain fusion result 1. Fusion result 1 is multiplied by value feature 1 to obtain the third feature information corresponding to task 1 (i.e., the updated first feature information). The execution device inputs the third feature information corresponding to task 1 into task head 1 (Head 1) and obtains prediction result 1 corresponding to task 1 output by Head 1; Head 1 is a feature processing network corresponding to task 1 among the three feature processing networks included in the first machine learning model.
[0154] Correspondingly, the execution device fuses the attention matrix 2 and the attention matrix 2 to obtain the fusion result 2, and multiplies the fusion result 2 by the value feature 2 to obtain the third feature information corresponding to task 2 (that is, the updated first feature information). The execution device inputs the third feature information corresponding to task 2 into the task head 2 (Head 2) to obtain the prediction result 2 corresponding to task 2 output by Head 2; Head 2 is a feature processing network corresponding to task 2 among the three feature processing networks included in the first machine learning model. It should be understood that the example in Figure 8 is only for the convenience of understanding this solution and is not used to limit this solution.
[0155] In an embodiment of the present application, the second feature information corresponding to other tasks (i.e., the second task) is integrated into the first feature information corresponding to the first task to obtain updated first feature information, which is beneficial for the feature processing network corresponding to the first task to obtain richer information, and thus is beneficial for improving the accuracy of the prediction results output by the first machine learning model.
[0156] 403. Perform feature processing on each third feature information of the first data through the first machine learning model to obtain at least one prediction result output by the first machine learning model that corresponds one-to-one to at least one task.
[0157] In an embodiment of the present application, the first machine learning model may include N feature processing networks corresponding one-to-one to N tasks. After the execution device obtains M third feature information of the first data through the feature extraction network of the first machine learning model, exemplarily, the process of the execution device processing the third feature information corresponding to one of the M tasks (hereinafter referred to as the "target task" for the convenience of description) may include: the execution device performs feature processing on the third feature information corresponding to the target task through a feature processing network corresponding to the target task among the N feature processing networks, and obtains a prediction result output by a feature processing network corresponding to the target task.
[0158] The execution device can perform the above operations respectively through M feature processing networks in N feature processing networks corresponding to M tasks, and obtain M prediction results output by the aforementioned M feature processing networks, that is, obtain M prediction results corresponding to M tasks.
[0159] Among them, each feature extraction network in the N feature extraction networks may include one or more neural network layers; optionally, at least one feature extraction network in the N feature extraction networks (hereinafter referred to as the "first feature extraction network" for the convenience of description) may have one or more first neural network layers, and the parameters of the first neural network layer are determined based on the first information, that is, in the process of feature processing using the first feature extraction network, the parameters adopted by each first neural network layer in the first feature extraction network will be determined based on the first information; the process of "determining the parameters adopted by the first neural network layer based on the first information" can refer to the description in the above step 402, and will not be repeated in the embodiment of the present application.
[0160] In order to understand the present solution more intuitively, please refer to Figure 9, which is a schematic diagram of the first machine learning model provided in an embodiment of the present application. As shown in Figure 9, the feature extraction network of the first machine learning model may include a first feature extraction stage and a second feature extraction stage when performing feature extraction on the input first data. The first feature extraction stage may include n convolution modules, and the feature information of the first data generated in the first feature extraction stage is feature information shared by M tasks; the specific implementation method of the "first feature extraction stage" can refer to the above description of Figure 6 and will not be repeated here.
[0161] The second feature extraction stage may include N adapters corresponding one to one to the N tasks (i.e., adapter 1, adapter 2, and adapter 3 in FIG6 ); each adapter is used to generate the first feature information corresponding to each of the M tasks based on the shared feature information, and then the updated first feature information (i.e., the third feature information) corresponding to each of the M tasks is generated through the information fusion module based on the attention mechanism; the specific implementation method of the "second feature extraction stage" can be found in the above description of FIG8 and will not be repeated here.
[0162] Since the first information indicates that Task 1 is to be executed on the input first data, after obtaining the third feature information corresponding to Task 1, the third feature information corresponding to Task 1 is feature processed by Task Head 1 corresponding to Task 1 to obtain a prediction result output by Task Head 1. As shown in FIG9 , the first machine learning model includes at least one first neural network layer in both the first feature extraction stage and the second feature extraction stage. It should be understood that the example in FIG6 is only for facilitating the understanding of this solution and is not intended to limit this solution.
[0163] In an embodiment of the present application, first information is added to the input of the first machine learning model, and the first information is used to indicate that at least one task needs to be performed on the first data. The first machine learning model outputs at least one prediction result that corresponds one-to-one to the at least one task. That is, the first machine learning model can know which tasks need to be performed on the first data based on the first information, and then can adaptively perform the required tasks on the first data, avoiding the generation of redundant prediction results, thereby avoiding wasting computer resources.
[0164] In an embodiment of the present application, in order to understand the present solution more intuitively, the following describes the network structure of the neural network of the execution device after obtaining the first data and the first information, using the first machine learning model to execute the first feature extraction stage in combination with Figures 10 to 12. In Figures 10 to 12, the first machine learning model uses a residual network (ResNet) as an example when executing the first feature extraction stage. First, refer to Figure 10, which is a schematic diagram of the residual network provided in the embodiment of the present application. Figure 10 shows that the ResNet in the first machine learning model includes 4 stages, the feature information in the same stage has the same resolution, and the feature information in different stages has different resolutions. In Figure 10, stage1, stage2, and stage4 each include a residual block, and stage3 includes 3 residual blocks as an example. After the first data input into the first machine learning model passes through stage1 to stage4 in the first machine learning model, the shared feature information of the first data is obtained.
[0165] Please continue to refer to Figure 11, which is a structural diagram of each residual block provided in an embodiment of the present application. Each residual block includes two sets of convolutional layers - batch normalization (BN) layer - rectified linear unit (ReLU) layer structures, and also includes a shortcut from the input of the residual block to the last ReLU layer; a shortcut refers to a jump connection across multiple neural network layers in the residual block. Among them, the convolutional layer and BN layer in each residual block are both the first neural network layer, that is, the parameters used by the convolutional layer and BN layer in each residual block are related to the first information.
[0166] Please refer to Figure 12. Figure 12 is a schematic diagram of the neural network used when performing the first feature extraction stage through the feature extraction network of the first machine learning model provided in an embodiment of the present application. As shown in Figure 12, on the basis of the residual network, additional first information is added as the input of the first machine learning model. The first information in Figure 12 is used to indicate that task 1 of the three tasks needs to be performed on the first data; each stage is configured with a corresponding first module, and stage1, stage2 and stage4 are each configured with a first module. Since there are three residual blocks in stage3, three first modules are configured for stage3. The multiple first modules in Figure 12 are all different first modules; the first module is used to generate the second parameter corresponding to each first neural network layer based on the feature information of the first information. The specific working process of the first module can be found in the above description and will not be repeated here. It should be understood that the example in Figure 12 is only for the convenience of understanding this solution and is not used to limit this solution.
[0167] 2. Training Phase
[0168] In the embodiment of the present application, the training phase describes the process in which the training device 210 performs a training operation on the first machine learning model 201 using the training data set. Specifically, please refer to FIG13 , which is a flow chart of a model training method provided in the embodiment of the present application. The model training method provided in the embodiment of the present application may include:
[0169] 1301. Acquire first data and first information, where the first information indicates at least one task performed on the first data.
[0170] 1302. Input the first data and the first information into a first machine learning model, process the first data through the first machine learning model, and obtain at least one prediction result output by the first machine learning model that corresponds one-to-one to at least one task.
[0171] In the embodiment of the present application, the first machine learning model has the ability to simultaneously perform N tasks on the input first data, and the first information indicates the M tasks performed on the first data, where N is an integer greater than or equal to 2, and M is an integer greater than or equal to 1. The specific implementation of steps 1301 and 1302 when the training device performs them, as well as the meanings of the terms in steps 1301 and 1302, can be found in the description of steps 401 to 403 in the embodiment corresponding to FIG. 4 , and will not be repeated here.
[0172] 1303. Train a first machine learning model based on at least one correct result and at least one predicted result corresponding to at least one task and a first loss function, wherein the first loss function indicates the similarity between the predicted result and the correct result corresponding to each task in the at least one task.
[0173] In an embodiment of the present application, in one implementation method, all parameters used in the first machine learning model are trained together. Then, in each training process, after obtaining the M prediction results output by the first machine learning model corresponding one-to-one to M tasks, the training device can generate a function value of the first loss function based on the M prediction results corresponding one-to-one to the M tasks and the M correct results; perform gradient differentiation on the first loss function, and use the backpropagation algorithm to update the parameter values in the first machine learning model to complete one training of the first machine learning model.
[0174] The first loss function indicates the similarity between the predicted result and the correct result corresponding to each of the M tasks, and the purpose of training using the first loss function includes improving the similarity between the predicted result and the correct result corresponding to each of the M tasks.
[0175] The training device may repeatedly perform steps 1302 to 1303 to iteratively train the first machine learning model until a convergence condition is satisfied, thereby obtaining a trained first machine learning model. For example, the convergence condition may be satisfying a convergence condition of the first loss function, or training the first machine learning model a preset number of times, etc., and these are not exhaustive.
[0176] In another implementation, if the first machine learning model includes a first module, the first module is used to generate the second parameters of each first neural network layer; optionally, the training process of the first machine learning model can be divided into a first training stage and a second training stage; in the first training stage, the training device can only update the parameters of the neural network layer outside the first module in the first machine learning model.
[0177] Exemplarily, in each training session of the first training phase, the first information indicates that all N tasks are to be performed on the first data, and the at least one prediction result corresponding one-to-one to the at least one task includes N prediction results corresponding one-to-one to the N tasks. The training device can generate a function value of a first loss function based on the N prediction results corresponding one-to-one to the N tasks and the N correct results; perform gradient differentiation on the first loss function, and use a backpropagation algorithm to update the parameters of the neural network layer outside the first module in the first machine learning model to complete one training of the first machine learning model. The first loss function indicates the similarity between the prediction result and the correct result corresponding to each of the N tasks.
[0178] The training device can repeatedly execute steps 1302 to 1303 to implement iterative training of the first machine learning model until the first convergence condition is met. The first convergence condition can be the convergence condition of the first loss function, or it can be the number of times the parameters of the neural network layer outside the first module in the first machine learning model are updated reaches a first preset number, etc., which are not enumerated here.
[0179] In each training of the second training stage, the first information indicates that M tasks out of N tasks are to be performed on the first data, and the at least one prediction result corresponding one-to-one to the at least one task includes M prediction results corresponding one-to-one to the M tasks. The specific implementation method in which the training device trains the first module in the first machine learning model according to the M prediction results corresponding one-to-one to the M tasks, the M correct results, and the first loss function can refer to the description in the previous implementation method, with the difference that, in the previous implementation method, the training device updates all parameters in the first machine learning model according to the function value of the first loss function; in this implementation method, since the parameters of the neural network layer outside the first module in the first machine learning model have been obtained in the first training stage, in the second training stage, only the parameters of each first module in the first machine learning model are updated until the second convergence condition is met.
[0180] Exemplarily, the first convergence condition may be satisfying the convergence condition of the first loss function, or it may be that the number of times the parameters of the first module in the first machine learning model are updated reaches a second preset number, etc., which are not enumerated here.
[0181] After completing the operations of the first training stage and the second training stage, the training device can obtain a first machine learning model that has performed training operations.
[0182] To further understand the beneficial effects of the embodiments of this application, the following experimental data is used to illustrate the results. Table 1 shows an example of an experiment conducted on the NYUDv2 dataset, with N tasks including semantic segmentation, depth estimation, and normal estimation.
[0183] Table 1
[0184] SingleTask represents the use of independent machine learning models to process N tasks respectively; BaseMultiTask represents the shared feature extraction network of N tasks, and the aforementioned shared feature extraction network is connected to N feature processing networks, thereby outputting the prediction results of N tasks respectively; the numbers in Table 1 represent the comprehensive performance of multiple tasks, and the comprehensive performance of the prediction results of N tasks obtained by using SingleTask is 0, that is, the comprehensive performance of the prediction results of N tasks obtained by using SingleTask is used as a reference benchmark; the trained machine learning model obtained by using BaseMultiTask reduces the comprehensive performance of the prediction results of N tasks; the trained machine learning model obtained by using the method provided in this application improves the comprehensive performance of the prediction results of N tasks.
[0185] On the basis of the embodiments corresponding to Figures 1 to 13, in order to better implement the above-mentioned solutions of the embodiments of the present application, the following also provides related equipment for implementing the above-mentioned solutions. Please refer to Figure 14 in detail. Figure 14 is a structural diagram of a data processing device provided in the embodiment of the present application. The data processing device 1400 includes: an acquisition module 1401, which is used to acquire first data and first information, and the first information indicates at least one task to be performed on the first data; a processing module 1402, which is used to input the first data and the first information into a first machine learning model, and process the first data through the first machine learning model to obtain at least one prediction result output by the first machine learning model that corresponds to at least one task.
[0186] Optionally, the first machine learning model includes multiple neural network layers, and there is at least one first neural network layer among the multiple neural network layers; wherein, the processing module 1402 is specifically used to: determine the parameters adopted by the first neural network layer based on the first information; according to the parameters adopted by the first neural network layer, use the first neural network layer to process the second data to obtain the processing result of the second data generated by the first neural network layer, where the second data is the first data or feature information of the first data.
[0187] Optionally, if the second data is the characteristic information of the first data, the processing result of the second data is the updated characteristic information of the first data; if the second data is the first data, the processing result of the second data is the characteristic information of the first data.
[0188] Optionally, the processing module 1402 is specifically used to: obtain a first parameter corresponding to the first neural network layer; determine a second parameter corresponding to the first neural network layer based on characteristic information of the first information; and determine a parameter adopted by the first neural network layer based on the first parameter and the second parameter.
[0189] Optionally, the processing module 1402 is specifically used to fuse the first feature information of the first data with the second feature information of the first data to obtain updated first feature information, and the updated first feature information is used to obtain a first prediction result, and the first prediction result is one of the at least one prediction results corresponding to the first task; wherein the first feature information corresponds to the first task, the second feature information corresponds to the second task, the first task is any one of the at least one task, and the second task is a task other than the first task in the at least one task.
[0190] Optionally, the processing module 1402 is specifically configured to fuse the first feature information with the second feature information based on an attention mechanism to obtain updated first feature information.
[0191] Optionally, the processing module 1402 is specifically used to: generate a first query feature, a first key feature and a first value feature based on the first feature information corresponding to the first task, and generate a first attention matrix corresponding to the first task based on the first query feature and the first key feature; obtain a second attention matrix corresponding to the second task, and the second attention matrix is obtained based on the second feature information corresponding to the second task; fuse the first attention matrix and the second attention matrix to obtain a fusion result; and generate an updated first feature information based on the fusion result and the first value feature.
[0192] Optionally, the first data is an image, and the at least one task includes any one or more of the following: image classification, target detection on an image, semantic segmentation on an image, segmenting an attention target from an image, text recognition on an image, image instance segmentation, posture estimation of a human body in an image, or action recognition of a human body in an image.
[0193] It should be noted that the information interaction, execution process, etc. between the modules / units in the data processing device 1400 are based on the same concept as the various method embodiments corresponding to Figures 3 to 12 in this application. For specific contents, please refer to the description in the method embodiments shown above in this application, and will not be repeated here.
[0194] Please continue to refer to Figure 15, which is a structural diagram of a model training device provided in an embodiment of the present application. The model training device 1500 includes: an acquisition module 1501, used to acquire first data and first information, the first information indicating at least one task performed on the first data; a processing module 1502, used to input the first data and the first information into a first machine learning model, process the first data through the first machine learning model, and obtain at least one prediction result output by the first machine learning model that corresponds one-to-one to at least one task; a training module 1503, used to train the first machine learning model based on at least one correct result and at least one prediction result corresponding one-to-one to at least one task and a loss function, the loss function indicating the similarity between the prediction result and the correct result corresponding to each task in at least one task.
[0195] Optionally, the first machine learning model includes multiple neural network layers, and there is at least one first neural network layer among the multiple neural network layers; wherein, the processing module 1502 is specifically used to: determine the parameters adopted by the first neural network layer based on the characteristic information of the first information; according to the parameters adopted by the first neural network layer, use the first neural network layer to process the second data to obtain the processing result of the second data generated by the first neural network layer, where the second data is the first data or the characteristic information of the first data.
[0196] Optionally, the processing module 1502 is specifically used to fuse the first feature information of the first data with the second feature information of the first data to obtain updated first feature information, and the updated first feature information is used to obtain a first prediction result, and the first prediction result is one of the at least one prediction results corresponding to the first task; wherein the first feature information corresponds to the first task, the second feature information corresponds to the second task, the first task is any one of the at least one task, and the second task is a task other than the first task in the at least one task.
[0197] It should be noted that the information interaction, execution process, etc. between the modules / units in the model training device 1500 are based on the same concept as the various method embodiments corresponding to Figure 13 in this application. For specific contents, please refer to the description in the method embodiments shown above in this application, and will not be repeated here.
[0198] Next, an execution device provided in an embodiment of the present application will be described. Please refer to FIG16 , which is a schematic structural diagram of an execution device provided in an embodiment of the present application. Specifically, the execution device 1600 includes: a receiver 1601, a transmitter 1602, a processor 1603, and a memory 1604 (wherein the number of processors 1603 in the execution device 1600 may be one or more, and FIG16 uses one processor as an example). The processor 1603 may include an application processor 16031 and a communication processor 16032. In some embodiments of the present application, the receiver 1601, the transmitter 1602, the processor 1603, and the memory 1604 may be connected via a bus or other means.
[0199] Memory 1604 may include read-only memory and random access memory, and provides instructions and data to processor 1603. A portion of memory 1604 may also include non-volatile random access memory (NVRAM). Memory 1604 stores processor and operation instructions, executable modules, or data structures, or subsets or extended sets thereof. The operation instructions may include various operation instructions for implementing various operations.
[0200] Processor 1603 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together via a bus system. In addition to a data bus, the bus system may also include a power bus, a control bus, and a status signal bus. However, for clarity, all bus systems are referred to as a bus system in the figure.
[0201] The methods disclosed in the above embodiments of the present application can be applied to or implemented by processor 1603. Processor 1603 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits in processor 1603 or software instructions. The above processor 1603 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and can further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor 1603 can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in conjunction with the embodiments of the present application can be directly implemented and executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium well-known in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, etc. The storage medium is located in memory 1604, and processor 1603 reads the information in memory 1604 and, in conjunction with its hardware, completes the steps of the above method.
[0202] Receiver 1601 can be used to receive input digital or character information and generate signal input related to executing device-related settings and function control. Transmitter 1602 can be used to output digital or character information through the first interface. Transmitter 1602 can also be used to send instructions to the disk pack through the first interface to modify data in the disk pack. Transmitter 1602 can also include a display device such as a display screen.
[0203] In the embodiment of the present application, processor 1603 is configured to execute the data processing method executed by the execution device in the embodiments corresponding to Figures 3 to 12. The specific manner in which the application processor 16031 in processor 1603 executes the aforementioned steps is based on the same concept as the various method embodiments corresponding to Figures 3 to 12 of the present application, and the technical effects brought about are the same as the various method embodiments corresponding to Figures 3 to 12 of the present application. For specific details, please refer to the description of the method embodiments shown above in the present application, and will not be repeated here.
[0204] The present application also provides a training device. Please refer to FIG. 17 , which is a schematic diagram of the structure of a training device provided by an embodiment of the present application. Specifically, the training device 1700 is implemented by one or more servers. The training device 1700 may vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 1722 (e.g., one or more processors) and a memory 1732, and one or more storage media 1730 (e.g., one or more mass storage devices) storing application programs 1742 or data 1744. The memory 1732 and storage medium 1730 may be either ephemeral or persistent storage. The program stored in the storage medium 1730 may include one or more modules (not shown), each of which may include a series of instruction operations on the training device. Furthermore, the CPU 1722 may be configured to communicate with the storage medium 1730 to execute the series of instruction operations in the storage medium 1730 on the training device 1700.
[0205] The training device 1700 may also include one or more power supplies 1726, one or more wired or wireless network interfaces 1750, one or more input and output interfaces 1758, and / or one or more operating systems 1741, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.
[0206] In the embodiment of the present application, the central processing unit 1722 is configured to execute the data processing method performed by the training device in the embodiment corresponding to FIG13 . The specific manner in which the central processing unit 1722 executes the aforementioned steps is based on the same concept as the various method embodiments corresponding to FIG13 in the present application, and the technical effects achieved are the same as the various method embodiments corresponding to FIG13 in the present application. For details, please refer to the description of the method embodiments shown above in the present application, and will not be repeated here.
[0207] A computer-readable storage medium is also provided in an embodiment of the present application. The computer-readable storage medium stores a program for signal processing. When the computer-readable storage medium is run on a computer, the computer executes the steps executed by the execution device in the method described in the embodiments shown in Figures 3 to 12 above, or the computer executes the steps executed by the training device in the method described in the embodiment shown in Figure 13 above.
[0208] Also provided in an embodiment of the present application is a computer program product, which, when executed on a computer, enables the computer to execute the steps executed by the execution device in the method described in the embodiments shown in Figures 3 to 12 above, or enables the computer to execute the steps executed by the training device in the method described in the embodiment shown in Figure 13 above.
[0209] The execution device or training device provided in the embodiments of the present application may specifically be a chip, and the chip includes: a processing unit and a communication unit, wherein the processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, a pin or a circuit, etc. The processing unit may execute the computer execution instructions stored in the storage unit, so that the chip executes the data processing method described in the embodiments shown in Figures 3 to 12 above, or so that the chip executes the model training method described in the embodiment shown in Figure 13 above. Optionally, the storage unit is a storage unit within the chip, such as a register, a cache, etc. The storage unit may also be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.
[0210] Specifically, see Figure 18 , which is a schematic diagram of the structure of a chip provided in an embodiment of the present application. The chip can be implemented as a neural network processor (NPU) 180. NPU 180 is mounted on a host CPU (host CPU) as a coprocessor, with tasks assigned by the host CPU. The core of the NPU is arithmetic circuit 1803, which is controlled by controller 1804 to extract matrix data from memory and perform multiplication operations.
[0211] In some implementations, the arithmetic circuit 1803 includes multiple processing units (PEs). In some implementations, the arithmetic circuit 1803 is a two-dimensional systolic array. The arithmetic circuit 1803 can also be a one-dimensional systolic array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1803 is a general-purpose matrix processor.
[0212] For example, assume there are input matrix A, weight matrix B, and output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from weight memory 1802 and caches it on each PE in the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from input memory 1801 and performs a matrix operation on matrix B. The partial or final matrix result is stored in accumulator 1808.
[0213] Unified memory 1806 is used to store input and output data. Weight data is directly transferred to weight memory 1802 through the Direct Memory Access Controller (DMAC) 1805. Input data is also transferred to unified memory 1806 through the DMAC.
[0214] BIU stands for Bus Interface Unit 1810 , which is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 1809 .
[0215] The bus interface unit 1810 (BIU) is used for the instruction fetch memory 1809 to obtain instructions from the external memory, and is also used for the storage unit access controller 1805 to obtain the original data of the input matrix A or the weight matrix B from the external memory.
[0216] DMAC is mainly used to move input data in the external memory DDR to the unified memory 1806 or to move weight data to the weight memory 1802 or to move input data to the input memory 1801.
[0217] The vector calculation unit 1807 includes multiple operation processing units. When necessary, it further processes the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.
[0218] In some implementations, the vector calculation unit 1807 can store the processed output vector to the unified memory 1806. For example, the vector calculation unit 1807 can apply a linear function and / or a nonlinear function to the output of the operation circuit 1803, such as linear interpolation of the feature plane extracted by the convolution layer, or accumulate a vector of values to generate an activation value. In some implementations, the vector calculation unit 1807 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1803, for example, for use in a subsequent layer in a neural network.
[0219] An instruction fetch buffer 1809 connected to the controller 1804 is used to store instructions used by the controller 1804;
[0220] Unified memory 1806, input memory 1801, weight memory 1802, and instruction fetch memory 1809 are all on-chip memories. External memories are private to the NPU hardware architecture.
[0221] Among them, the operations of each layer in the high-dimensional convolutional neural network shown in Figure 7 and Figure 7 can be performed by the operation circuit 1803 or the vector calculation unit 1807.
[0222] The processor mentioned in any of the above places can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the program of the above-mentioned first aspect method.
[0223] It should also be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.
[0224] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course can also be implemented by special hardware including application-specific integrated circuits, special CPUs, special memories, special components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits or special circuits, etc. However, for the present application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a number of instructions to enable a computer device (which can be a personal computer, training equipment, or network equipment, etc.) to execute the methods described in each embodiment of the present application.
[0225] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.
[0226] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a training device or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website, a computer, a training device or a data center. The computer-readable storage medium can be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium can be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).
Claims
1. A data processing method, characterized in that: The method comprises: Acquire first data and first information, wherein the first information indicates at least one task performed on the first data; The first data and the first information are input into a first machine learning model, and the first data is processed by the first machine learning model to obtain at least one prediction result output by the first machine learning model that corresponds one-to-one to the at least one task.
2. The method according to claim 1, characterized in that The first machine learning model includes a plurality of neural network layers, wherein the plurality of neural network layers include at least one first neural network layer, wherein the processing of the first data by using the first machine learning model includes: Determining parameters used by the first neural network layer based on the first information; According to the parameters adopted by the first neural network layer, the first neural network layer is used to process the second data to obtain a processing result of the second data generated by the first neural network layer, where the second data is the first data or feature information of the first data.
3. The method according to claim 2, characterized in that If the second data is the characteristic information of the first data, the processing result of the second data is the updated characteristic information of the first data; if the second data is the first data, the processing result of the second data is the characteristic information of the first data.
4. The method according to claim 2, characterized in that: Determining the parameters used by the first neural network layer based on the first information includes: Obtaining a first parameter corresponding to the first neural network layer; Determining a second parameter corresponding to the first neural network layer according to the characteristic information of the first information; The parameters used by the first neural network layer are determined according to the first parameters and the second parameters.
5. The method according to any one of claims 1 to 4, characterized in that: The processing of the first data by using the first machine learning model includes: fusing the first feature information of the first data with the second feature information of the first data to obtain updated first feature information, wherein the updated first feature information is used to obtain a first prediction result, where the first prediction result is one of the at least one prediction result corresponding to the first task; The first characteristic information corresponds to the first task, the second characteristic information corresponds to the second task, the first task is any one of the at least one task, and the second task is a task other than the first task in the at least one task.
6. The method according to claim 5, characterized in that The fusing the first feature information of the first data with the second feature information of the first data to obtain updated first feature information includes: Based on the attention mechanism, the first feature information is fused with the second feature information to obtain the updated first feature information.
7. The method according to claim 6, characterized in that The step of fusing the first feature information with the second feature information based on the attention mechanism to obtain the updated first feature information includes: Generate a first query feature, a first key feature, and a first value feature according to the first feature information corresponding to the first task, and generate a first attention matrix corresponding to the first task according to the first query feature and the first key feature; Acquire a second attention matrix corresponding to the second task, where the second attention matrix is obtained according to the second feature information corresponding to the second task; Fusing the first attention matrix and the second attention matrix to obtain a fusion result; Based on the fusion result and the first value feature, the updated first feature information is generated.
8. The method according to any one of claims 1 to 4, characterized in that: The first data is an image, and the at least one task includes any one or more of the following: image classification, target detection on an image, semantic segmentation on an image, segmenting an attention target from an image, text recognition on an image, image instance segmentation, posture estimation of a human body in an image, or action recognition of a human body in an image.
9. A model training method, characterized in that: The method comprises: Acquire first data and first information, wherein the first information indicates at least one task performed on the first data; Inputting the first data and the first information into a first machine learning model, processing the first data by the first machine learning model, and obtaining at least one prediction result output by the first machine learning model corresponding to the at least one task; The first machine learning model is trained based on at least one correct result corresponding to the at least one task, the at least one predicted result, and a loss function, wherein the loss function indicates the similarity between the predicted result and the correct result corresponding to each task in the at least one task.
10. The method according to claim 9, characterized in that The first machine learning model includes a plurality of neural network layers, wherein the plurality of neural network layers include at least one first neural network layer, wherein the processing of the first data by using the first machine learning model includes: Determining parameters used by the first neural network layer according to the characteristic information of the first information; According to the parameters adopted by the first neural network layer, the first neural network layer is used to process the second data to obtain a processing result of the second data generated by the first neural network layer, where the second data is the first data or feature information of the first data.
11. The method according to claim 9 or 10, characterized in that: The processing of the first data by using the first machine learning model includes: fusing the first feature information of the first data with the second feature information of the first data to obtain updated first feature information, wherein the updated first feature information is used to obtain a first prediction result, where the first prediction result is one of the at least one prediction result corresponding to the first task; The first characteristic information corresponds to the first task, the second characteristic information corresponds to the second task, the first task is any one of the at least one task, and the second task is a task other than the first task in the at least one task.
12. A data processing device, characterized in that: The device comprises: an acquisition module, configured to acquire first data and first information, wherein the first information indicates at least one task performed on the first data; A processing module is used to input the first data and the first information into a first machine learning model, process the first data through the first machine learning model, and obtain at least one prediction result output by the first machine learning model that corresponds one-to-one to the at least one task.
13. The device according to claim 12, characterized in that The first machine learning model includes a plurality of neural network layers, wherein the plurality of neural network layers includes at least one first neural network layer; Wherein, the processing module is specifically used for: Determining parameters used by the first neural network layer based on the first information; According to the parameters adopted by the first neural network layer, the first neural network layer is used to process the second data to obtain a processing result of the second data generated by the first neural network layer, where the second data is the first data or feature information of the first data.
14. The device according to claim 13, characterized in that If the second data is the characteristic information of the first data, the processing result of the second data is the updated characteristic information of the first data; if the second data is the first data, the processing result of the second data is the characteristic information of the first data.
15. The device according to claim 13, characterized in that The processing module is specifically used for: Obtaining a first parameter corresponding to the first neural network layer; Determining a second parameter corresponding to the first neural network layer according to the characteristic information of the first information; The parameters used by the first neural network layer are determined according to the first parameters and the second parameters.
16. The device according to any one of claims 12 to 15, characterized in that The processing module is specifically used to fuse the first feature information of the first data with the second feature information of the first data to obtain updated first feature information, wherein the updated first feature information is used to obtain a first prediction result, and the first prediction result is one of the at least one prediction result corresponding to the first task; The first characteristic information corresponds to the first task, the second characteristic information corresponds to the second task, the first task is any one of the at least one task, and the second task is a task other than the first task in the at least one task.
17. The device according to claim 16, characterized in that The processing module is specifically used to fuse the first feature information with the second feature information based on an attention mechanism to obtain the updated first feature information.
18. The device according to claim 17, characterized in that The processing module is specifically used for: Generate a first query feature, a first key feature, and a first value feature according to the first feature information corresponding to the first task, and generate a first attention matrix corresponding to the first task according to the first query feature and the first key feature; Acquire a second attention matrix corresponding to the second task, where the second attention matrix is obtained according to the second feature information corresponding to the second task; Fusing the first attention matrix and the second attention matrix to obtain a fusion result; Based on the fusion result and the first value feature, the updated first feature information is generated.
19. The device according to any one of claims 12 to 15, characterized in that The first data is an image, and the at least one task includes any one or more of the following: image classification, target detection on an image, semantic segmentation on an image, segmenting an attention target from an image, text recognition on an image, image instance segmentation, posture estimation of a human body in an image, or action recognition of a human body in an image.
20. A model training device, characterized in that: The device comprises: an acquisition module, configured to acquire first data and first information, wherein the first information indicates at least one task performed on the first data; a processing module, configured to input the first data and the first information into a first machine learning model, process the first data through the first machine learning model, and obtain at least one prediction result output by the first machine learning model corresponding to the at least one task; A training module is used to train the first machine learning model according to at least one correct result corresponding to the at least one task and the at least one predicted result and a loss function, wherein the loss function indicates the similarity between the predicted result and the correct result corresponding to each task in the at least one task.
21. The device according to claim 20, characterized in that The first machine learning model includes a plurality of neural network layers, wherein the plurality of neural network layers includes at least one first neural network layer; Wherein, the processing module is specifically used for: Determining parameters used by the first neural network layer according to the characteristic information of the first information; According to the parameters adopted by the first neural network layer, the first neural network layer is used to process the second data to obtain a processing result of the second data generated by the first neural network layer, where the second data is the first data or feature information of the first data.
22. The device according to claim 20 or 21, characterized in that The processing module is specifically used to fuse the first feature information of the first data with the second feature information of the first data to obtain updated first feature information, wherein the updated first feature information is used to obtain a first prediction result, and the first prediction result is one of the at least one prediction result corresponding to the first task; The first characteristic information corresponds to the first task, the second characteristic information corresponds to the second task, the first task is any one of the at least one task, and the second task is a task other than the first task in the at least one task.
23. An execution device, characterized in that: comprising a processor and a memory, the processor being coupled to the memory, The memory is used to store programs; The processor is configured to execute the program in the memory so that the execution device executes the method according to any one of claims 1 to 8.
24. A training device, characterized in that comprising a processor and a memory, the processor being coupled to the memory, The memory is used to store programs; The processor is used to execute the program in the memory so that the training device performs the method as claimed in any one of claims 9 to 11.
25. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program, and when the program is executed on a computer, the computer is caused to execute the method according to any one of claims 1 to 11.
26. A computer program product, characterized in that The computer program product comprises a program, and when the program is run on a computer, the computer is caused to execute the method according to any one of claims 1 to 11.
Citation Information
Patent Citations
Method for processing multiple tasks, related apparatus and computer program product
CN113378855A
Network equipment, data transmission method, device and system and readable storage medium
CN114640602A
Image processing method, neural network training method and related equipment
CN114821096A
Model prediction method and device, electronic equipment and storage medium
CN115543603A
Data processing method, model training method and related equipment
CN116882512A
