Multi-task recognition method, training method, device, electronic device and storage medium

Through the multi-task recognition method of deep learning model, the task probability value and category probability value are calculated using intermediate feature data, which solves the problems of cumbersome multi-task recognition process and large resource utilization in the existing technology, and achieves efficient multi-task recognition effect.

CN117315313BActive Publication Date: 2024-12-03BEIJING BAIDU NETCOM SCI & TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210335746.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-03-30
Publication Date
2024-12-03
Estimated Expiration
2042-03-30

AI Technical Summary

Technical Problem

When implementing multi-task recognition between users and vehicles, the prior art requires the deployment of multiple single models, resulting in cumbersome online process, large memory usage, high energy consumption, and it is difficult to ensure that the multi-task recognition accuracy is not lower than the single-task recognition accuracy.

Method used

Using multi-task recognition method, the image to be identified is converted into intermediate feature data through deep learning models, including auxiliary feature data and category feature data, calculate task probability values ​​and category probability values, and then obtain multi-task recognition results, simplify the online process, and reduce the Memory usage and reduces energy consumption.

Benefits of technology

The accuracy of multi-task recognition is achieved is not lower than that of single-task recognition, and at the same time, the online process is simplified, memory and energy consumption are reduced, and system efficiency and manageability are improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117315313B_ABST
    Figure CN117315313B_ABST
Patent Text Reader

Abstract

The present disclosure provides a multi-task recognition method, training method, device, electronic device and storage medium, which relate to the field of artificial intelligence technology, especially to the field of deep learning, image processing and computer vision technology, and can be applied to scenes such as faces. The specific implementation scheme is: according to the image to be recognized, first intermediate feature data is obtained, wherein the first intermediate feature data includes first auxiliary feature data and first category feature data, and the first auxiliary feature data is data used to assist the multi-task recognition of the image to be recognized; according to the first auxiliary feature data and the first category feature data, the task probability value and the category probability value of the image to be recognized are obtained; according to the task probability value and the category probability value of the image to be recognized, the multi-task recognition result of the image to be recognized is obtained.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the field of deep learning, image processing, and computer vision technology, and can be applied to scenes such as faces. Specifically, it relates to a multi-task recognition method, training method, device, electronic device, and storage medium. Background Art

[0002] With the development of computer technology, artificial intelligence technology has also developed. Artificial intelligence technology can include computer vision technology, speech recognition technology, natural language processing technology, machine learning, deep learning, big data processing technology and knowledge graph technology.

[0003] Artificial intelligence technology has been widely used in various fields. For example, artificial intelligence technology can be used to achieve multi-task recognition. Summary of the invention

[0004] The present disclosure provides a multi-task recognition method, a training method, a device, an electronic device, and a storage medium.

[0005] According to one aspect of the present disclosure, a multi-task recognition method is provided, comprising: obtaining first intermediate feature data according to an image to be recognized, wherein the first intermediate feature data comprises first auxiliary feature data and first category feature data, and the first auxiliary feature data is data for assisting multi-task recognition of the image to be recognized; obtaining a task probability value and a category probability value of the image to be recognized according to the first auxiliary feature data and the first category feature data; and obtaining a multi-task recognition result of the image to be recognized according to the task probability value and the category probability value of the image to be recognized.

[0006] According to another aspect of the present disclosure, a method for training a deep learning model is provided, comprising: obtaining first intermediate sample feature data based on a sample image, wherein the first intermediate sample feature data includes first sample auxiliary feature data and first sample category feature data, and the first sample auxiliary feature data is data for assisting multi-task recognition of the sample image; obtaining a sample task probability value and a sample category probability value of the sample image based on the first sample auxiliary feature data and the first sample category feature data; and training the deep learning model using the sample task probability value, the sample category probability value and the label value of the sample image to obtain a trained deep learning model.

[0007] According to another aspect of the present disclosure, a multi-task recognition device is provided, including: a first acquisition module, used to obtain first intermediate feature data according to an image to be recognized, wherein the first intermediate feature data includes first auxiliary feature data and first category feature data, and the first auxiliary feature data is data used to assist the multi-task recognition of the image to be recognized; a second acquisition module, used to obtain a task probability value and a category probability value of the image to be recognized according to the first auxiliary feature data and the first category feature data; and a third acquisition module, used to obtain a multi-task recognition result of the image to be recognized according to the task probability value and the category probability value of the image to be recognized.

[0008] According to an embodiment of the present disclosure, a training device for a deep learning model is provided, comprising:

[0009] A fourth acquisition module is used to obtain first intermediate sample feature data based on the sample image, wherein the first intermediate sample feature data includes first sample auxiliary feature data and first sample category feature data, and the first sample auxiliary feature data is data used to assist multi-task recognition of the sample image; a fifth acquisition module is used to obtain a sample task probability value and a sample category probability value of the sample image based on the first sample auxiliary feature data and the first sample category feature data; and a sixth acquisition module is used to train the deep learning model using the sample task probability value, sample category probability value and label value of the sample image to obtain a trained deep learning model.

[0010] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method described in the present disclosure.

[0011] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described in the present disclosure.

[0012] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the method described in the present disclosure is implemented.

[0013] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0015] Figure 1 An exemplary system architecture to which a multi-task recognition method, a training method and a device for a deep learning model can be applied according to an embodiment of the present disclosure is schematically shown;

[0016] Figure 2 A flowchart of a multi-task recognition method according to an embodiment of the present disclosure is schematically shown;

[0017] Figure 3 A flowchart schematically illustrates how to obtain first intermediate feature data according to an image to be recognized according to an embodiment of the present disclosure. The first intermediate feature data includes first auxiliary feature data and first category feature data;

[0018] Figure 4 An example schematic diagram of a multi-task recognition method according to an embodiment of the present disclosure is schematically shown;

[0019] Figure 5 A flowchart of a deep learning model training method according to an embodiment of the present disclosure is schematically shown;

[0020] Figure 6 An example schematic diagram of a deep learning model according to an embodiment of the present disclosure is schematically shown;

[0021] Figure 7 An example schematic diagram schematically illustrates a method for training a deep learning model according to an embodiment of the present disclosure;

[0022] Figure 8 A block diagram of a multi-task recognition device according to an embodiment of the present disclosure is schematically shown;

[0023] Fig. 9 A block diagram schematically shows a training device for a deep learning model according to an embodiment of the present disclosure; and

[0024] Fig.10 A block diagram of an electronic device suitable for implementing a multi-task recognition method and a deep learning model training method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0025] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0026] The core function of smart cities is the accurate positioning and intelligent management of users and vehicles. In practical applications, in order to realize the recognition function for users and vehicles, multiple single models need to be deployed for single-task recognition of users and vehicles. The above method is not only cumbersome in the online process, for example, it is necessary to train and test the single model corresponding to each task, but also multiple single models will occupy more memory and increase energy consumption.

[0027] To this end, a multi-task recognition method can be used to simplify the online process, reduce memory usage and reduce energy consumption. Multi-task recognition can refer to the ability to recognize multiple tasks. The use of multi-task recognition requires that the multi-task recognition accuracy is not lower than the single-task recognition accuracy.

[0028] Figure 1 An exemplary system architecture to which a multi-task recognition method, a training method and a device for a deep learning model can be applied according to an embodiment of the present disclosure is schematically illustrated.

[0029] It should be noted that Figure 1 The examples shown are only examples of system architectures to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios. For example, in another embodiment, an exemplary system architecture to which the multi-task recognition method, the training method of the deep learning model and the device can be applied may include a terminal device, but the terminal device may implement the multi-task recognition method, the training method of the deep learning model and the device provided in the embodiments of the present disclosure without interacting with the server.

[0030] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0031] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients and / or social platform software, etc. (only as examples).

[0032] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0033] Server 105 can be any type of server that provides various services. For example, server 105 can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services (Virtual Private Server). Server 105 can also be a server of a distributed system, or a server combined with a blockchain.

[0034] It should be noted that the multi-task identification method provided in the embodiment of the present disclosure can generally be executed by the terminal device 101, 102, or 103. Accordingly, the multi-task identification apparatus provided in the embodiment of the present disclosure can also be set in the terminal device 101, 102, or 103.

[0035] Alternatively, the multi-task identification method provided in the embodiment of the present disclosure may also be generally performed by the server 105. Accordingly, the multi-task identification device provided in the embodiment of the present disclosure may generally be arranged in the server 105. The multi-task identification method provided in the embodiment of the present disclosure may also be performed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the multi-task identification device provided in the embodiment of the present disclosure may also be arranged in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0036] It should be noted that the training method of the deep learning model provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the training device of the deep learning model provided in the embodiment of the present disclosure can generally be set in the server 105. The training method of the deep learning model provided in the embodiment of the present disclosure can also be executed by a server or a server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the training device of the deep learning model provided in the embodiment of the present disclosure can also be set in a server or a server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0037] Alternatively, the training method of the deep learning model provided in the embodiment of the present disclosure may also be generally performed by the terminal device 101, 102, or 103. Accordingly, the training apparatus of the deep learning model provided in the embodiment of the present disclosure may also be provided in the terminal device 101, 102, or 103.

[0038] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0039] It should be noted that the sequence numbers of the operations in the following method are only used as representations of the operations for the purpose of description, and should not be regarded as representing the execution order of the operations. Unless explicitly stated, the method does not need to be executed completely in the order shown.

[0040] Figure 2 The flowchart of the multi-task recognition method according to an embodiment of the present disclosure is schematically shown.

[0041] like Figure 2 As shown, the method 200 includes operations S210 to S230.

[0042] In operation S210, first intermediate feature data is obtained according to the image to be recognized, the first intermediate feature data including first auxiliary feature data and first category feature data, and the first auxiliary feature data is data for assisting multi-task recognition of the image to be recognized.

[0043] In operation S220, a task probability value and a category probability value of the image to be recognized are obtained according to the first auxiliary feature data and the first category feature data.

[0044] In operation S230 , a multi-task recognition result of the image to be recognized is obtained according to the task probability value and the category probability value of the image to be recognized.

[0045] According to an embodiment of the present disclosure, an image to be identified may refer to an image that needs to be multi-task identified. Multi-task identification may refer to image recognition for multiple tasks. Multi-task identification may include at least two of the following: living body recognition, signboard recognition, obstacle recognition, building recognition, and vehicle recognition. Living body recognition may include at least one of the following: face recognition and body recognition. Body recognition may include body motion and behavior recognition. An image to be identified may include at least one of the following: a living body image to be identified, a signboard image to be identified, an obstacle image to be identified, a building image to be identified, and a vehicle image to be identified. A living body image to be identified may include at least one of the following: a face image to be identified and a body image to be identified.

[0046] According to an embodiment of the present disclosure, auxiliary feature data may refer to data used to assist in multi-task recognition of an image to be recognized. Auxiliary feature data may include data related to multiple recognition tasks. For example, the auxiliary feature data may include task feature data. Task feature data may refer to data of a task dimension. For example, task feature data may include task classification feature data. Task classification feature data may refer to data related to an image in a task dimension. Category feature data may refer to data related to an image in a category dimension.

[0047] According to an embodiment of the present disclosure, the task dimension may include at least one of the following: living body recognition, sign recognition, obstacle recognition, building recognition, and vehicle recognition. The category dimension may include at least one of the following: at least one category related to living body recognition, at least one category related to sign recognition, at least one category related to obstacle recognition, at least one category related to building recognition, and at least one category related to vehicle recognition. The at least one category related to living body recognition may include at least one of the following: at least one category related to face recognition and at least one category related to body recognition.

[0048] For example, at least one category related to face recognition may include at least one of the following: an elderly category, a middle-aged category, a young person category, a teenager category, a toddler category, and an infant category. At least one category related to human body recognition may include at least one of the following: a walking action category and a sports action category. At least one category related to vehicle recognition may include at least one of the following: a passenger car category and a commercial vehicle category. The passenger car category may include at least one of the following: a basic passenger car category, a multi-purpose vehicle category, a sports utility vehicle category, and other vehicle category. The commercial vehicle category may include at least one of the following: a passenger car category, a truck category, a semi-trailer category, a passenger incomplete vehicle category, and a cargo incomplete vehicle category.

[0049] According to an embodiment of the present disclosure, the task probability value of an image to be identified may refer to the probability value of each of at least one task to which the image to be identified belongs. The category probability value of an image to be identified may refer to the probability value of each of at least one category to which the image to be identified belongs. For example, when the image to be identified is a face image to be identified, and the task dimensions include face recognition, body recognition, and vehicle recognition, the task probability value of the image to be identified may refer to the probability values ​​of each of face recognition, body recognition, and vehicle recognition to which the face image to be identified belongs. The category probability value of an image to be identified may refer to the probability value of at least one category related to face recognition to which the face image to be identified belongs.

[0050] According to an embodiment of the present disclosure, a multi-task recognition result can characterize a target category of a target task to which an image to be recognized belongs. The target task can be the task with the highest probability among multiple tasks. The target category can be the category with the highest probability among multiple categories. The probability can be characterized by a probability value. The relationship between the probability and the probability value can be configured according to actual business needs and is not limited here. For example, the larger the probability value, the higher the probability. Alternatively, the smaller the probability value, the higher the probability.

[0051] According to an embodiment of the present disclosure, an image to be identified can be processed to obtain first intermediate feature data. The first intermediate feature data may include first auxiliary feature data and first category feature data. In addition, the first intermediate feature data may also include first image feature data. Based on the first auxiliary feature data, a task probability value of the image to be identified can be obtained. Based on the first category feature data, a category probability value of the image to be identified can be obtained. Based on the task probability value and the category probability value of the image to be identified, a multi-task recognition result representing the target category of the target task to which the image to be identified belongs is determined.

[0052] According to an embodiment of the present disclosure, operations S210 to S230 may be performed by an electronic device. The electronic device may be a terminal device or a server. The terminal device may be Figure 1 The server may be a terminal device 101, a terminal device 102 or a terminal device 103. Figure 1 Server 105 in.

[0053] According to an embodiment of the present disclosure, the multi-task recognition task for the image to be recognized is assisted by introducing the first auxiliary feature data. The first auxiliary feature data is task-related data, and therefore, the multi-task recognition accuracy can be effectively guaranteed.

[0054] According to an embodiment of the present disclosure, the first auxiliary feature data may include first task feature data.

[0055] According to an embodiment of the present disclosure, the second auxiliary feature data may include second task feature data.

[0056] According to an embodiment of the present disclosure, both the first task feature data and the second task feature data may be data for assisting in multi-task recognition of an image to be recognized.

[0057] Reference below Figure 3-4 , the multi-task recognition method according to the embodiment of the present disclosure is further explained in combination with specific embodiments.

[0058] Figure 3 The flowchart of obtaining first intermediate feature data according to an image to be recognized according to an embodiment of the present disclosure is schematically shown. The first intermediate feature data includes first auxiliary feature data and first category feature data.

[0059] like Figure 3 As shown, the method 300 is to Figure 2 As a further limitation of operation S210, the method 300 includes operations S311-S312.

[0060] In operation S311, second intermediate feature data is obtained according to the image to be recognized and predetermined data, and the second intermediate feature data includes second auxiliary feature data, second category feature data, and object feature data of each of a plurality of image blocks to be recognized in the image to be recognized.

[0061] In operation S312 , at least one of a global feature and a local feature of the second intermediate feature data is extracted to obtain first intermediate feature data.

[0062] According to an embodiment of the present disclosure, predetermined data may refer to data related to generating second auxiliary feature data and second category feature data. An image may include multiple image blocks. The image blocks may be obtained by dividing the image. The size of the image blocks may be configured according to actual business needs and is not limited here. The sizes of different image blocks may be the same. Object feature data may refer to feature data of an image block. For an image to be identified, the image to be identified may include multiple image blocks to be identified. The image blocks to be identified may be obtained by dividing the image to be identified. The sizes of different image blocks to be identified may be the same.

[0063] According to an embodiment of the present disclosure, an image to be identified and predetermined data may be obtained. The image to be identified may be processed to obtain a plurality of image blocks to be identified. The plurality of image blocks to be identified may be processed to obtain object feature data of each of the plurality of image blocks to be identified. The predetermined data may be processed to obtain second auxiliary feature data and second category feature data.

[0064] According to an embodiment of the present disclosure, after obtaining the second intermediate feature data, feature extraction of at least one of a global feature and a local feature may be performed on the second intermediate feature data to obtain the first intermediate feature data. That is, global feature extraction may be performed on the second intermediate feature data to obtain the first intermediate feature data. Alternatively, local feature extraction may be performed on the second intermediate feature data to obtain the first intermediate feature data. Alternatively, feature extraction of global features and local features may be performed on the second intermediate feature data to obtain the first intermediate feature data.

[0065] According to an embodiment of the present disclosure, operations S311 to S312 may be performed by an electronic device. The electronic device may be a terminal device or a server. The terminal device may be Figure 1 The server may be a terminal device 101, a terminal device 102 or a terminal device 103. Figure 1 Server 105 in.

[0066] According to an embodiment of the present disclosure, the predetermined data includes first predetermined data and second predetermined data.

[0067] According to an embodiment of the present disclosure, operation S311 may include the following operations.

[0068] The image to be identified is processed to obtain object feature data of each of the multiple image blocks to be identified included in the image to be identified. The first predetermined data is processed to obtain second auxiliary feature data. The second predetermined data is processed to obtain second category feature data. The second intermediate feature data is obtained according to the second auxiliary feature data, the second category feature data and the object feature data of each of the multiple image blocks to be identified.

[0069] According to an embodiment of the present disclosure, the first predetermined data may refer to data related to generating the second auxiliary feature data. The second predetermined data may refer to data related to generating the second category feature data.

[0070] According to an embodiment of the present disclosure, feature extraction can be performed on the first predetermined data to obtain second auxiliary feature data. Feature extraction can be performed on the second predetermined data to obtain second category feature data. Feature extraction is performed on the image to be identified to obtain object feature data of each of the multiple image blocks to be identified. The second auxiliary feature data, the second category feature data and the object feature data of each of the multiple images to be identified are spliced ​​to obtain second intermediate feature data. For example, the second auxiliary feature data and the second category feature data can be set at a predetermined position and spliced ​​with the object feature data of each of the multiple image blocks to be identified to obtain second intermediate feature data. The predetermined position can be configured according to actual business needs and is not limited here. For example, multiple image blocks to be identified of the image to be identified can form a sequence of image blocks to be identified. The predetermined position can be a position before the starting position of the sequence of image blocks to be identified. Alternatively, the predetermined position can be a position after the ending position of the sequence of image blocks to be identified.

[0071] According to an embodiment of the present disclosure, the second auxiliary feature data may be obtained by processing the first predetermined data using an auxiliary processing unit of a deep learning model.

[0072] According to an embodiment of the present disclosure, the deep learning model may include a preprocessing module. The preprocessing module may include an auxiliary processing unit. The auxiliary processing unit may include a network structure that can be used to implement feature extraction. For example, the auxiliary processing unit may include a convolutional neural network.

[0073] According to an embodiment of the present disclosure, the second predetermined data may be processed by an auxiliary processing unit to obtain second auxiliary feature data.

[0074] According to an embodiment of the present disclosure, operation S312 may include the following operations.

[0075] Based on the attention strategy and the local strategy, the second intermediate feature data is processed to obtain the first intermediate feature data.

[0076] According to an embodiment of the present disclosure, the attention strategy can be used to focus on important information with high weights and ignore unimportant information with low weights, and can exchange information with other information by sharing important information, thereby realizing the transmission of important information. The local feature strategy can be used to realize the extraction of local features of objects. In an embodiment of the present disclosure, the attention strategy can extract information from the second auxiliary feature data itself, the second category feature data itself, the inside of the image block to be identified, and the second auxiliary feature data, the second category feature data, and the image block to be identified, so as to better complete the processing of the image to be identified. The local strategy can be used to realize the extraction of information from the second auxiliary feature data itself, the second category feature data itself, and the inside of the image block to be identified.

[0077] According to an embodiment of the present disclosure, the second intermediate feature data can be processed based on the attention strategy and the local strategy to obtain the first intermediate feature data for characterizing the global features and local features of the image to be identified. For example, the attention layer can be determined according to the attention strategy. The feedforward network layer can be determined according to the local strategy. The output of the attention layer can be used as the input of the feedforward neural network layer. The second intermediate feature data is processed using the attention layer and the feedforward network layer to obtain the first intermediate feature data.

[0078] According to an embodiment of the present disclosure, the first intermediate feature data includes first auxiliary feature data, and the second intermediate feature data includes second auxiliary feature data. The first intermediate feature data is obtained by processing the second intermediate feature data based on the attention strategy and the local strategy. Therefore, the first auxiliary feature data participates in the global self-attention mechanism and couples the task information, thereby improving the multi-task recognition accuracy.

[0079] According to an embodiment of the present disclosure, the first intermediate feature data may be obtained by processing the second intermediate feature data using a backbone module of a deep learning model. The backbone module includes at least one cascaded backbone unit. The backbone unit includes a multi-head self-attention layer and a feedforward network layer.

[0080] According to an embodiment of the present disclosure, the backbone unit may include a multi-head self-attention (MHA) layer and a feed forward network (FFN). The backbone module may include a Transformer.

[0081] According to an embodiment of the present disclosure, the backbone module may be used to extract global features and local features of the second intermediate feature data to obtain the first intermediate feature data.

[0082] According to an embodiment of the present disclosure, the backbone module may include M backbone units connected in cascade, where M is an integer greater than or equal to 1.

[0083] According to an embodiment of the present disclosure, processing the second intermediate feature data using a backbone module of a deep learning model to obtain the first intermediate feature data may include the following operations.

[0084] In the case of M>1 and m>1,

[0085] The fourth intermediate feature data of the (m-1)th level is processed by the multi-head self-attention layer of the mth level to obtain the third intermediate feature data of the mth level. The third intermediate feature data of the mth level is processed by the feedforward network layer of the mth level to obtain the fourth intermediate feature data of the mth level. The first intermediate feature data is obtained according to the fourth intermediate feature data of the Nth level. N is an integer greater than 1 and less than or equal to M.

[0086] According to an embodiment of the present disclosure, M may be an integer greater than or equal to 1. N may be an integer greater than 1 and less than or equal to M. The values ​​of M and N may be configured according to actual business requirements and are not limited here. For example, M=N=4. m∈{1, 2, ..., (M-1), M}.

[0087] According to an embodiment of the present disclosure, when M=1, the multi-head self-attention layer of the first layer can be used to process the second intermediate feature data to obtain the third intermediate feature data of the first layer. The third intermediate feature data of the first layer is processed by the feedforward network layer of the first layer to obtain the fourth intermediate feature data of the first layer. The first intermediate feature data is obtained according to the fourth intermediate feature data of the first layer. For example, the fourth intermediate feature data of the first layer can be determined as the first intermediate feature data.

[0088] According to an embodiment of the present disclosure, the fourth intermediate feature data of the Nth level may be determined as the first intermediate feature data. N may be equal to M.

[0089] According to an embodiment of the present disclosure, the object feature data may be obtained by processing the image block using an object processing unit of a deep learning model.

[0090] According to an embodiment of the present disclosure, the second category feature data may be obtained by processing the second predetermined data using a category processing unit of a deep learning model.

[0091] According to an embodiment of the present disclosure, the deep learning model may include a preprocessing module. The preprocessing module may include an object processing unit and a category processing unit. The object processing unit may be used to process a plurality of image blocks to be identified included in the image to be identified, and obtain object feature data of each of the plurality of image blocks to be identified. The category processing unit may be used to process the second predetermined data, and obtain second category feature data. Both the object processing unit and the category processing unit may include a network structure that can be used to implement feature extraction. For example, the object processing unit may include a convolutional neural network. The category processing unit may include a convolutional neural network. The network structures of the object processing unit and the category processing unit may be the same or different.

[0092] According to an embodiment of the present disclosure, the task probability value may be obtained by processing the first auxiliary feature data using at least one auxiliary classification unit of the deep learning model.

[0093] According to an embodiment of the present disclosure, the category probability value is obtained by processing the first category feature data using at least one category classification unit of the deep learning model.

[0094] According to an embodiment of the present disclosure, the deep learning model may include a classification module. The classification module may include at least one auxiliary classification unit and at least one category classification unit. The auxiliary classification unit may be used to process the first auxiliary feature data to obtain a task probability value. The category classification unit may be used to process the first category feature data to obtain a category probability value. Both the object processing unit and the category processing unit may include a network structure that can be used to implement classification. For example, both the auxiliary classification unit and the category classification unit may include one of a linear classifier and a nonlinear classifier. The network structures of the auxiliary classification unit and the category classification unit may be the same or different.

[0095] Figure 4 An example schematic diagram of a multi-task recognition method according to an embodiment of the present disclosure is schematically shown.

[0096] like Figure 4 As shown, in 400 , an image 401 to be recognized, first predetermined data 403 , and second predetermined data 405 may be obtained.

[0097] The image to be recognized 401 is processed to obtain object feature data 402 of each of the plurality of image blocks to be recognized included in the image to be recognized 401 .

[0098] The first predetermined data 403 is processed to obtain second task characteristic data 404 .

[0099] The second predetermined data 405 is processed to obtain second category characteristic data 406 .

[0100] Second intermediate feature data 407 is obtained according to the second task feature data 404 , the second category feature data 406 and the object feature data 402 of each of the plurality of image blocks to be identified.

[0101] Based on the attention strategy and the local strategy, the second intermediate feature data 407 is processed to obtain the first intermediate feature data 408. The first intermediate feature data 308 includes the first task feature data 409 and the first category feature data 410.

[0102] According to the first task feature data 409, a task probability value 411 of the image to be recognized 401 is obtained. According to the first category feature data 410, a category probability value 412 of the image to be recognized 401 is obtained.

[0103] According to the task probability value 411 and the category probability value 412 of the image to be recognized 401 , a multi-task recognition result 413 of the image to be recognized 401 is obtained.

[0104] Figure 5 A flowchart of a deep learning model training method according to an embodiment of the present disclosure is schematically shown.

[0105] like Figure 5 As shown, the method 500 includes operations S510 to S530.

[0106] In operation S510, first intermediate sample feature data is obtained according to the sample image. The first intermediate sample feature data includes first sample auxiliary feature data and first sample category feature data. The first sample auxiliary feature data is data used to assist multi-task recognition of the sample image.

[0107] In operation S520, a sample task probability value and a sample category probability value of the sample image are obtained according to the first sample auxiliary feature data and the first sample category feature data.

[0108] In operation S530, a deep learning model is trained using the sample task probability value, the sample category probability value, and the label value of the sample image to obtain a trained deep learning model.

[0109] According to the embodiments of the present disclosure, for the description of the sample image, the first intermediate sample feature data, the first sample auxiliary feature data and the first sample category feature data, please refer to the relevant content of the image to be identified, the first intermediate feature data, the first auxiliary feature data and the first category feature data above, which will not be repeated here.

[0110] According to an embodiment of the present disclosure, the deep learning model may include a preprocessing module, a backbone module and a classification module. The preprocessing module may include an auxiliary processing unit and a category processing unit. The auxiliary processing unit may be used to obtain the auxiliary feature data of the first sample. The category processing unit may be used to obtain the category feature data of the first sample.

[0111] According to an embodiment of the present disclosure, based on a loss function, a deep learning model is trained using sample task probability values, sample category probability values, and label values ​​of a sample image to obtain a trained deep learning model. The trained deep learning model can be used to implement multi-task recognition. The loss function can be configured according to actual business needs and is not limited here. For example, the loss function may include at least one of the following: a cross entropy loss function, an exponential loss function, and a square loss function. The predetermined condition may include at least one of output value convergence and training rounds reaching a maximum training round.

[0112] According to an embodiment of the present disclosure, operations S510 to S530 may be performed by an electronic device. The electronic device may be a server or a terminal device. The server may be Figure 1 The terminal device may be a server 105 in Figure 1 The terminal device 101, the terminal device 102 or the terminal device 103.

[0113] According to an embodiment of the present disclosure, the training process of the auxiliary supervision deep learning model is realized by introducing the first sample auxiliary feature data, which reduces the training difficulty of the deep learning model. In addition, the first sample auxiliary feature data is task-related data, and therefore, the multi-task recognition accuracy of the trained deep learning model can be effectively guaranteed.

[0114] According to an embodiment of the present disclosure, the first sample auxiliary feature data includes first sample task feature data.

[0115] According to an embodiment of the present disclosure, the second sample auxiliary feature data includes second sample task feature data.

[0116] According to an embodiment of the present disclosure, a label value includes an auxiliary label value and a category label value.

[0117] According to an embodiment of the present disclosure, operation S530 may include the following operations.

[0118] Based on the first loss function, a first output value is obtained using the sample task probability value and the auxiliary label value. Based on the second loss function, a second output value is obtained using the sample category probability value and the category label value. The output value is obtained according to the first output value and the second output value. The model parameters of the deep learning model are adjusted according to the output value until a predetermined end condition is met, thereby obtaining a trained deep learning model.

[0119] According to an embodiment of the present disclosure, the loss function may be determined according to a first loss function and a second loss function. The first loss function and the second loss function may include a cross entropy loss function. The auxiliary label value may include a task label value.

[0120] According to an embodiment of the present disclosure, the sample task probability value and the auxiliary label value may be input into a first loss function to obtain a first output value. The sample category probability value and the category label value may be input into a second loss function to obtain a second output value. The output value may be obtained based on the first output value and the second output value. For example, a weighted sum value of the first output value and the second output value may be determined. The weighted sum value is determined as the output value.

[0121] According to an embodiment of the present disclosure, the model parameters of the deep learning model can be adjusted according to the output value until a predetermined end condition is met. The deep learning model obtained when the predetermined end condition is met can be determined as a trained deep learning model. The predetermined end condition may include at least one of the following: the output value converges and the training rounds reach the maximum training rounds. For example, the model parameters of the deep learning model can be adjusted according to the back propagation algorithm or the stochastic gradient descent algorithm until the predetermined end condition is met.

[0122] According to an embodiment of the present disclosure, the first loss function may be determined according to the following formula (1).

[0123]

[0124] According to an embodiment of the present disclosure, L t represents the first loss function. B represents the number of sample images included in each batch. B can be an integer greater than 1. T represents the number of tasks. T can be a number greater than 1. y ij Characterizes the task label value of the jth task of the i-th sample image, that is, y ij Indicates whether the i-th sample image belongs to the j-th task. ij =1 indicates that the i-th sample image belongs to the j-th task. ij = 0 indicates that the i-th sample image does not belong to the j-th task. ij Characterizes the task probability value of the jth task of the i-th sample image, that is, p ij Represents the probability value that the i-th sample image belongs to the j-th task.

[0125] According to an embodiment of the present disclosure, the second loss function may be determined according to the following formula (2).

[0126]

[0127] According to an embodiment of the present disclosure, L jCharacterizes the second loss function. C j Indicates the number of categories included in the jth task. j Can be an integer greater than or equal to 1. ijk Characterizes the category label value of the kth category of the jth task of the ith sample image, that is, z ijk Indicates whether the i-th sample image belongs to the k-th category of the j-th task. ijk =1 indicates that the i-th sample image belongs to the k-th category of the j-th task. ijk = 0 indicates that the i-th sample image does not belong to the k-th category of the j-th task. ijk Characterizes the category probability value of the kth category of the jth task of the ith sample image, that is, p ijk The probability value that represents the i-th sample image belongs to the k-th category of the j-th task.

[0128] According to an embodiment of the present disclosure, the loss function can be determined according to the following formula (3).

[0129]

[0130] According to an embodiment of the present disclosure, L represents the loss function. t Representation and the first loss function L t The corresponding weighting coefficient. t The value can be configured according to actual business needs and is not limited here.

[0131] According to an embodiment of the present disclosure, operation S510 may include the following operations.

[0132] According to the sample image and the predetermined sample data, second intermediate sample feature data is obtained. The second intermediate sample feature data includes second sample auxiliary feature data, second sample category feature data, and sample object feature data of each of a plurality of sample image blocks in the sample image. At least one of the global feature and the local feature of the second intermediate sample feature data is extracted to obtain the first intermediate sample feature data.

[0133] According to an embodiment of the present disclosure, the predetermined sample data may include first predetermined sample data and second predetermined sample data.

[0134] According to an embodiment of the present disclosure, obtaining second intermediate sample feature data according to a sample image and predetermined sample data may include the following operations.

[0135] The sample image is processed to obtain sample object feature data of each of the multiple sample image blocks included in the sample image. The first predetermined sample data is processed to obtain second sample auxiliary feature data. The second predetermined sample data is processed to obtain second sample category feature data. The second intermediate sample feature data is obtained according to the second sample auxiliary feature data, the second sample category feature data and the sample object feature data of each of the multiple sample image blocks.

[0136] According to an embodiment of the present disclosure, the deep learning model may include an auxiliary processing unit.

[0137] According to an embodiment of the present disclosure, processing the first predetermined sample data to obtain the second sample auxiliary feature data may include the following operations.

[0138] The first predetermined sample data is processed by an auxiliary processing unit to obtain second sample auxiliary feature data.

[0139] According to an embodiment of the present disclosure, extracting at least one of the global feature and the local feature of the second intermediate sample feature data to obtain the first intermediate sample feature data may include the following operations.

[0140] Based on the attention strategy and the local strategy, the second intermediate sample feature data is processed to obtain the first intermediate sample feature data.

[0141] According to an embodiment of the present disclosure, the deep learning model may include a backbone module. The backbone module may include at least one backbone unit in cascade. The backbone unit may include a multi-head self-attention layer and a feedforward network layer.

[0142] According to an embodiment of the present disclosure, processing the second intermediate sample feature data based on the attention strategy and the local strategy to obtain the first intermediate sample feature data may include the following operations.

[0143] The second intermediate sample characteristic data is processed by the backbone module to obtain the first intermediate sample characteristic data.

[0144] According to an embodiment of the present disclosure, the first intermediate sample feature data includes first sample auxiliary feature data, and the second intermediate sample feature data includes second sample auxiliary feature data. The first intermediate sample feature data is obtained by processing the second intermediate sample feature data using a backbone unit including a multi-head self-attention layer. Therefore, the first sample auxiliary feature data participates in the global self-attention mechanism and couples the task information, thereby improving the multi-task recognition accuracy.

[0145] According to an embodiment of the present disclosure, the backbone submodule may include M backbone units connected in cascade, where M is an integer greater than or equal to 1.

[0146] According to an embodiment of the present disclosure, using a backbone module to process the second intermediate sample feature data to obtain the first intermediate sample feature data may include the following operations.

[0147] In the case of M>1 and m>1,

[0148] Process the fourth intermediate sample feature data of the (m-1)th level using the multi-head self-attention layer of the mth level to obtain the third intermediate sample feature data of the mth level;

[0149] The third intermediate sample feature data of the m-th level is processed by the feedforward network layer of the m-th level to obtain the fourth intermediate sample feature data of the m-th level.

[0150] The first intermediate sample feature data is obtained according to the fourth intermediate sample feature data of the Nth level. N is an integer greater than 1 and less than or equal to M.

[0151] According to an embodiment of the present disclosure, when M=1, the multi-head self-attention layer of the first layer can be used to process the second intermediate sample feature data to obtain the third intermediate sample feature data of the first layer. The third intermediate sample feature data of the first layer is processed by the feedforward network layer of the first layer to obtain the fourth intermediate sample feature data of the first layer. The first intermediate sample feature data is obtained according to the fourth intermediate sample feature data of the first layer. For example, the fourth intermediate sample feature data of the first layer can be determined as the first intermediate sample feature data.

[0152] According to an embodiment of the present disclosure, sample object feature data is obtained by processing sample image blocks using an object processing unit of a deep learning model.

[0153] According to an embodiment of the present disclosure, the second sample category feature data is obtained by processing the second predetermined sample data using a category processing unit of a deep learning model.

[0154] According to an embodiment of the present disclosure, the sample task probability value is obtained by processing the first sample auxiliary feature data using the auxiliary classification unit of the deep learning model.

[0155] According to an embodiment of the present disclosure, the sample category probability value is obtained by processing the first sample category feature data using a category classification unit of a deep learning model.

[0156] Reference below Figure 6 and Figure 7 , the training method of the deep learning model according to the embodiment of the present disclosure is further explained in combination with specific embodiments.

[0157] Figure 6 An example schematic diagram of a deep learning model according to an embodiment of the present disclosure is schematically shown.

[0158] like Figure 6 As shown in 600 , the deep learning model 601 includes a preprocessing module 602 , a backbone module 603 , and a classification module 604 .

[0159] The pre-processing module 602 may include an object processing unit 6020 , a task processing unit 6021 , and a category processing unit 6022 .

[0160] The backbone module 603 may include M cascaded backbone units, namely, backbone unit 603_1, ..., backbone unit 603_m, ..., backbone unit 603_M. M may be an integer greater than or equal to 1.

[0161] The classification module 604 may include a task classification unit 6040 and a category classification unit 6041 .

[0162] Figure 7 An example schematic diagram of a training method for a deep learning model according to an embodiment of the present disclosure is schematically shown.

[0163] like Figure 7 As shown, in 700, the deep learning model includes a preprocessing module, a backbone module 711 and a classification module. The preprocessing module includes an object processing unit 702, a task processing unit 705 and a category processing unit 708. The classification module includes a task classification unit 715 and a category classification unit 720. The backbone module may include a Transformer.

[0164] The sample image 701 is processed by the object processing unit 702 to obtain sample object feature data 703 of each of the multiple sample image blocks included in the sample image 701. The first predetermined sample data 704 is processed by the task processing unit 705 to obtain second sample task feature data 706. The second predetermined sample data 707 is processed by the category processing unit 708 to obtain second sample category feature data 609. According to the second sample task feature data 706, the second sample category feature data 709 and the sample object feature data 703 of each of the multiple sample image blocks, the second intermediate sample feature data 710 is obtained.

[0165] The second intermediate sample characteristic data 710 is processed by the backbone module 611 to obtain the first intermediate sample characteristic data 712. The first intermediate sample characteristic data 712 includes the first sample task characteristic data 713 and the first sample category characteristic data 714.

[0166] The first sample task feature data 713 is processed by the task classification unit 715 to obtain a sample task probability value 716. Based on the first loss function 718, the sample task probability value 716 and the task label value 717 are used to obtain a first output value 719.

[0167] The first sample category feature data 714 is processed by the category classification unit 720 to obtain a sample category probability value 721. Based on the second loss function 723, the sample category probability value 721 and the category label value 722 are used to obtain a second output value 724.

[0168] According to the first output value 719 and the second output value 724, an output value 725 is obtained. According to the output value 725, the model parameters of the object processing unit 702, the task processing unit 705, the category processing unit 708, the backbone module 711, the task classification unit 715 and the category classification unit 720 are adjusted until the predetermined end condition is met, and a trained deep learning model is obtained.

[0169] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0170] The above are only exemplary embodiments, but are not limited thereto and may also include other multi-task recognition methods and deep learning model training methods known in the art, as long as they can effectively ensure the multi-task recognition accuracy and reduce the difficulty of model training.

[0171] Figure 8 The block diagram of a multi-task identification device according to an embodiment of the present disclosure is schematically shown.

[0172] like Figure 8 As shown, the multi-task identification device 800 may include a first obtaining module 810 , a second obtaining module 820 and a third obtaining module 830 .

[0173] The first acquisition module 810 is used to obtain first intermediate feature data according to the image to be recognized. The first intermediate feature data includes first auxiliary feature data and first category feature data. The first auxiliary feature data is data used to assist in multi-task recognition of the image to be recognized.

[0174] The second obtaining module 820 is used to obtain the task probability value and the category probability value of the image to be identified according to the first auxiliary feature data and the first category feature data.

[0175] The third obtaining module 830 is used to obtain a multi-task recognition result of the image to be recognized according to the task probability value and the category probability value of the image to be recognized.

[0176] According to an embodiment of the present disclosure, the first obtaining module 810 may include a first obtaining submodule and a second obtaining submodule.

[0177] The first acquisition submodule is used to obtain second intermediate feature data according to the image to be identified and the predetermined data. The second intermediate feature data includes second auxiliary feature data, second category feature data and object feature data of each of a plurality of image blocks to be identified in the image to be identified.

[0178] The second obtaining submodule is used to extract at least one of the global feature and the local feature of the second intermediate feature data to obtain the first intermediate feature data.

[0179] According to an embodiment of the present disclosure, the predetermined data includes first predetermined data and second predetermined data.

[0180] According to an embodiment of the present disclosure, the first obtaining submodule may include a first obtaining unit, a second obtaining unit, a third obtaining unit, and a fourth obtaining unit.

[0181] The first obtaining unit is used to process the image to be identified to obtain object feature data of each of a plurality of image blocks to be identified included in the image to be identified.

[0182] The second obtaining unit is used to process the first predetermined data to obtain second auxiliary feature data.

[0183] The third obtaining unit is used to process the second predetermined data to obtain second category characteristic data.

[0184] The fourth obtaining unit is used to obtain second intermediate feature data according to the second auxiliary feature data, the second category feature data and the object feature data of each of the plurality of image blocks to be identified.

[0185] According to an embodiment of the present disclosure, the second obtaining submodule may include a fifth obtaining unit.

[0186] The fifth obtaining unit is used to process the second intermediate feature data based on the attention strategy and the local strategy to obtain the first intermediate feature data.

[0187] According to an embodiment of the present disclosure, the first intermediate feature data is obtained by processing the second intermediate feature data using a backbone module of a deep learning model. The backbone module includes at least one cascaded backbone unit. The backbone unit includes a multi-head self-attention layer and a feedforward network layer.

[0188] According to an embodiment of the present disclosure, the second auxiliary feature data is obtained by processing the first predetermined data using an auxiliary processing unit of a deep learning model. The auxiliary processing unit includes a convolutional neural network.

[0189] According to an embodiment of the present disclosure, the first auxiliary feature data includes first task feature data.

[0190] Fig. 9A block diagram of a deep learning model training device according to an embodiment of the present disclosure is schematically shown.

[0191] like Fig. 9 As shown, the deep learning model training device 900 may include a fourth acquisition module 910, a second acquisition module 920 and a third acquisition module 930.

[0192] The fourth obtaining module 910 is used to obtain first intermediate sample feature data according to the sample image. The first intermediate sample feature data includes first sample auxiliary feature data and first sample category feature data. The first sample auxiliary feature data is data used to assist in multi-task recognition of the sample image.

[0193] The fifth obtaining module 920 is used to obtain a sample task probability value and a sample category probability value of the sample image according to the first sample auxiliary feature data and the first sample category feature data.

[0194] The sixth acquisition module 930 is used to train the deep learning model using the sample task probability value, the sample category probability value and the label value of the sample image to obtain a trained deep learning model.

[0195] According to an embodiment of the present disclosure, the fourth obtaining module 910 may include a third obtaining submodule and a fourth obtaining submodule.

[0196] The third obtaining submodule is used to obtain second intermediate sample feature data according to the sample image and the predetermined sample data. The second intermediate sample feature data includes second sample auxiliary feature data, second sample category feature data and sample object feature data of each of the plurality of sample image blocks in the sample image.

[0197] The fourth obtaining submodule is used to extract at least one of the global feature and the local feature of the second intermediate sample feature data to obtain the first intermediate sample feature data.

[0198] According to an embodiment of the present disclosure, the predetermined sample data includes first predetermined sample data and second predetermined sample data.

[0199] According to an embodiment of the present disclosure, the third obtaining submodule may include a sixth obtaining unit, a seventh obtaining unit, an eighth obtaining unit, and a ninth obtaining unit.

[0200] The sixth obtaining unit is used to process the sample image to obtain sample object feature data of each of the multiple sample image blocks included in the sample image.

[0201] The seventh obtaining unit is used to process the first predetermined sample data to obtain second sample auxiliary feature data.

[0202] The eighth obtaining unit is used to process the second predetermined sample data to obtain second sample category characteristic data.

[0203] The ninth obtaining unit is used to obtain second intermediate sample feature data according to the second sample auxiliary feature data, the second sample category feature data and the sample object feature data of each of the plurality of sample image blocks.

[0204] According to an embodiment of the present disclosure, the deep learning model includes an auxiliary processing unit.

[0205] According to an embodiment of the present disclosure, the seventh obtaining unit may include a first obtaining submodule.

[0206] The first obtaining subunit is used to process the first predetermined sample data using the auxiliary processing unit to obtain second sample auxiliary feature data.

[0207] According to an embodiment of the present disclosure, the fourth obtaining submodule may include a tenth obtaining unit.

[0208] The tenth obtaining unit is used to process the second intermediate sample feature data based on the attention strategy and the local strategy to obtain the first intermediate sample feature data.

[0209] According to an embodiment of the present disclosure, the deep learning model includes a backbone module. The backbone module includes at least one cascaded backbone unit. The backbone unit includes a multi-head self-attention layer and a feedforward network layer.

[0210] According to an embodiment of the present disclosure, the tenth obtaining unit may include a second obtaining sub-unit.

[0211] The second obtaining subunit is used to process the second intermediate sample characteristic data using the backbone module to obtain the first intermediate sample characteristic data.

[0212] According to an embodiment of the present disclosure, the backbone submodule includes M backbone units connected in cascade, where M is an integer greater than or equal to 1.

[0213] According to an embodiment of the present disclosure, the second obtaining subunit may be used to:

[0214] In the case of M>1 and m>1,

[0215] The fourth intermediate sample feature data of the (m-1)th level is processed by the multi-head self-attention layer of the mth level to obtain the third intermediate sample feature data of the mth level.

[0216] The third intermediate sample feature data of the m-th level is processed by the feedforward network layer of the m-th level to obtain the fourth intermediate sample feature data of the m-th level.

[0217] The first intermediate sample feature data is obtained according to the fourth intermediate sample feature data of the Nth level. N is an integer greater than 1 and less than or equal to M.

[0218] According to an embodiment of the present disclosure, sample object feature data is obtained by processing sample image blocks using an object processing unit of a deep learning model.

[0219] According to an embodiment of the present disclosure, the second sample category feature data is obtained by processing the second predetermined sample data using a category processing unit of a deep learning model.

[0220] According to an embodiment of the present disclosure, the sample task probability value is obtained by processing the first sample auxiliary feature data using the auxiliary classification unit of the deep learning model.

[0221] According to an embodiment of the present disclosure, the sample category probability value is obtained by processing the first sample category feature data using a category classification unit of a deep learning model.

[0222] According to an embodiment of the present disclosure, a label value includes an auxiliary label value and a category label value.

[0223] According to an embodiment of the present disclosure, the sixth obtaining module 930 may include a fifth obtaining sub-module, a sixth obtaining sub-module, a seventh obtaining sub-module, and an eighth obtaining sub-module.

[0224] The fifth acquisition submodule is used to obtain a first output value based on the first loss function using the sample task probability value and the auxiliary label value.

[0225] The sixth acquisition submodule is used to obtain a second output value based on the second loss function using the sample category probability value and the category label value.

[0226] The seventh obtaining submodule is used to obtain an output value according to the first output value and the second output value.

[0227] The eighth acquisition submodule is used to adjust the model parameters of the deep learning model according to the output value until a predetermined end condition is met to obtain a trained deep learning model.

[0228] According to an embodiment of the present disclosure, the first sample auxiliary feature data includes first sample task feature data.

[0229] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0230] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.

[0231] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method as described above.

[0232] According to an embodiment of the present disclosure, a computer program product includes a computer program, and when the computer program is executed by a processor, the computer program implements the method as described above.

[0233] Fig.10 A block diagram of an electronic device suitable for implementing a multi-task recognition method and a training method for a deep learning model according to an embodiment of the present disclosure is schematically shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0234] like Fig.10 As shown, the electronic device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the electronic device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0235] Multiple components in the electronic device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the electronic device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0236] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as a multi-task recognition method and a training method for a deep learning model. For example, in some embodiments, the multi-task recognition method and the training method for a deep learning model may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1000 via ROM 1002 and / or the communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of the multi-task recognition method and the training method for a deep learning model described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute the multi-task recognition method and the deep learning model training method in any other appropriate manner (e.g., by means of firmware).

[0237] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0238] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0239] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0240] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0241] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0242] A computer system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0243] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0244] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A multi-task recognition method, include: Obtaining first intermediate feature data according to the image to be identified, wherein the first intermediate feature data includes first auxiliary feature data and first category feature data, and the first auxiliary feature data is data used to assist multi-task identification of the image to be identified; According to the first auxiliary feature data and the first category feature data, a task probability value and a category probability value of the image to be identified are obtained, wherein the task probability value is obtained by processing the first auxiliary feature data using at least one auxiliary classification unit of the deep learning model, and the category probability value is obtained by processing the first category feature data using at least one category classification unit of the deep learning model. The task probability value includes a probability value of the image to be identified belonging to at least one task, and the category probability value includes a probability value of the image to be identified belonging to at least one category; and A multi-task recognition result of the image to be recognized is obtained according to the task probability value and the category probability value of the image to be recognized.

2. The method according to claim 1, in, The step of obtaining first intermediate feature data according to the image to be recognized includes: Obtaining second intermediate feature data according to the image to be identified and predetermined data, wherein the second intermediate feature data includes second auxiliary feature data, second category feature data, and object feature data of each of a plurality of image blocks to be identified in the image to be identified; and At least one of a global feature and a local feature of the second intermediate feature data is extracted to obtain the first intermediate feature data.

3. The method according to claim 2, in, The predetermined data includes first predetermined data and second predetermined data; Wherein, obtaining the second intermediate feature data according to the image to be recognized and the predetermined data includes: Processing the image to be identified to obtain object feature data of each of a plurality of image blocks to be identified included in the image to be identified; Processing the first predetermined data to obtain the second auxiliary feature data; Processing the second predetermined data to obtain the second category characteristic data; and The second intermediate feature data is obtained according to the second auxiliary feature data, the second category feature data and the object feature data of each of the plurality of image blocks to be identified.

4. The method according to claim 2, in, The extracting at least one of the global feature and the local feature of the second intermediate feature data to obtain the first intermediate feature data includes: Based on the attention strategy and the local strategy, the second intermediate feature data is processed to obtain the first intermediate feature data.

5. The method according to claim 4, in, The first intermediate feature data is obtained by processing the second intermediate feature data using the backbone module of the deep learning model, wherein the backbone module includes at least one cascaded backbone unit, and the backbone unit includes a multi-head self-attention layer and a feedforward network layer.

6. The method according to claim 3, in, The second auxiliary feature data is obtained by processing the first predetermined data using an auxiliary processing unit of the deep learning model, wherein the auxiliary processing unit includes a convolutional neural network.

7. The method according to any one of claims 1 to 6, in, The first auxiliary feature data includes first task feature data.

8. A training method for a deep learning model, include: According to the sample image, first intermediate sample feature data is obtained, wherein the first intermediate sample feature data includes first sample auxiliary feature data and first sample category feature data, and the first sample auxiliary feature data is data used to assist multi-task recognition of the sample image; A sample task probability value and a sample category probability value of the sample image are obtained according to the first sample auxiliary feature data and the first sample category feature data, wherein the sample task probability value is obtained by processing the first sample auxiliary feature data using at least one auxiliary classification unit of the deep learning model, and the sample category probability value is obtained by processing the sample first category feature data using at least one category classification unit of the deep learning model. The sample task probability value includes the probability value of the sample image belonging to at least one task, and the sample category probability value includes the probability value of the sample image belonging to at least one category; and The deep learning model is trained using the sample task probability value, the sample category probability value and the label value of the sample image to obtain a trained deep learning model.

9. The method according to claim 8, in, The step of obtaining first intermediate sample feature data according to the sample image includes: According to the sample image and the predetermined sample data, second intermediate sample feature data is obtained, wherein the second intermediate sample feature data includes second sample auxiliary feature data, second sample category feature data, and sample object feature data of each of a plurality of sample image blocks in the sample image; and At least one of the global feature and the local feature of the second intermediate sample feature data is extracted to obtain the first intermediate sample feature data.

10. The method according to claim 9, in, The predetermined sample data includes first predetermined sample data and second predetermined sample data; Wherein, obtaining the second intermediate sample feature data according to the sample image and the predetermined sample data includes: Processing the sample image to obtain sample object feature data of each of a plurality of sample image blocks included in the sample image; Processing the first predetermined sample data to obtain the second sample auxiliary feature data; Processing the second predetermined sample data to obtain the second sample category feature data; and The second intermediate sample feature data is obtained according to the second sample auxiliary feature data, the second sample category feature data and the sample object feature data of each of the plurality of sample image blocks.

11. The method according to claim 10, in, The deep learning model includes an auxiliary processing unit; The processing of the first predetermined sample data to obtain the second sample auxiliary feature data includes: The first predetermined sample data is processed by the auxiliary processing unit to obtain the second sample auxiliary feature data.

12. The method according to claim 9, in, The extracting at least one of the global feature and the local feature of the second intermediate sample feature data to obtain the first intermediate sample feature data includes: Based on the attention strategy and the local strategy, the second intermediate sample feature data is processed to obtain the first intermediate sample feature data.

13. The method according to claim 12, in, The deep learning model includes a backbone module, the backbone module includes at least one cascaded backbone unit, and the backbone unit includes a multi-head self-attention layer and a feedforward network layer; The processing of the second intermediate sample feature data based on the attention strategy and the local strategy to obtain the first intermediate sample feature data includes: The second intermediate sample characteristic data is processed by the backbone module to obtain the first intermediate sample characteristic data.

14. The method according to claim 13, in, The backbone module includes M cascaded backbone units, where M is an integer greater than or equal to 1; The step of processing the second intermediate sample feature data using the backbone module to obtain the first intermediate sample feature data includes: In the case of M>1 and m>1, The fourth intermediate sample feature data of the (m-1)th level is processed by the multi-head self-attention layer of the mth level to obtain the third intermediate sample feature data of the mth level; Processing the third intermediate sample feature data of the mth level by using the feedforward network layer of the mth level to obtain fourth intermediate sample feature data of the mth level; and The first intermediate sample feature data is obtained according to the fourth intermediate sample feature data of the Nth level, wherein N is an integer greater than 1 and less than or equal to M.

15. The method according to claim 10, in, The sample object feature data is obtained by processing the sample image block using the object processing unit of the deep learning model; The second sample category feature data is obtained by processing the second predetermined sample data using the category processing unit of the deep learning model.

16. The method according to any one of claims 8 to 15, in, The sample task probability value is obtained by processing the first sample auxiliary feature data using the auxiliary classification unit of the deep learning model; Among them, the sample category probability value is obtained by processing the first sample category feature data using the category classification unit of the deep learning model.

17. The method according to any one of claims 8 to 15, in, The label value includes an auxiliary label value and a category label value; The step of training the deep learning model using the sample task probability value, the sample category probability value and the label value of the sample image to obtain a trained deep learning model includes: Based on the first loss function, using the sample task probability value and the auxiliary label value, obtaining a first output value; Based on a second loss function, using the sample category probability value and the category label value, obtaining a second output value; Obtaining the output value according to the first output value and the second output value; and The model parameters of the deep learning model are adjusted according to the output value until a predetermined end condition is met, thereby obtaining the trained deep learning model.

18. The method according to any one of claims 8 to 15, in, The first sample auxiliary feature data includes first sample task feature data.

19. A multi-task recognition device, include: A first obtaining module, configured to obtain first intermediate feature data according to the image to be identified, wherein the first intermediate feature data includes first auxiliary feature data and first category feature data, and the first auxiliary feature data is data used to assist in multi-task identification of the image to be identified; a second obtaining module, configured to obtain a task probability value and a category probability value of the image to be identified according to the first auxiliary feature data and the first category feature data, wherein the task probability value is obtained by processing the first auxiliary feature data using at least one auxiliary classification unit of the deep learning model, and the category probability value is obtained by processing the first category feature data using at least one category classification unit of the deep learning model, The task probability value includes a probability value of the image to be identified belonging to at least one task, and the category probability value includes a probability value of the image to be identified belonging to at least one category; and The third obtaining module is used to obtain the multi-task recognition result of the image to be recognized according to the task probability value and the category probability value of the image to be recognized.

20. The device according to claim 19, in, The first obtaining module comprises: a first obtaining submodule, configured to obtain second intermediate feature data according to the image to be identified and predetermined data, wherein the second intermediate feature data includes second auxiliary feature data, second category feature data, and object feature data of each of a plurality of image blocks to be identified in the image to be identified; and The second obtaining submodule is used to extract at least one of the global feature and the local feature of the second intermediate feature data to obtain the first intermediate feature data.

21. The device according to claim 20, in, The predetermined data includes first predetermined data and second predetermined data; Wherein, the first obtaining submodule includes: A first obtaining unit is used to process the image to be identified to obtain object feature data of each of a plurality of image blocks to be identified included in the image to be identified; a second obtaining unit, configured to process the first predetermined data to obtain the second auxiliary feature data; A third obtaining unit is used to process the second predetermined data to obtain the second category characteristic data; and The fourth obtaining unit is used to obtain the second intermediate feature data according to the second auxiliary feature data, the second category feature data and the object feature data of each of the plurality of image blocks to be identified.

22. The device according to claim 20, in, The second obtaining submodule includes: The fifth obtaining unit is used to process the second intermediate feature data based on the attention strategy and the local strategy to obtain the first intermediate feature data.

23. The device according to claim 22, in, The first intermediate feature data is obtained by processing the second intermediate feature data using the backbone module of the deep learning model, wherein the backbone module includes at least one cascaded backbone unit, and the backbone unit includes a multi-head self-attention layer and a feedforward network layer.

24. The device according to any one of claims 20 to 23, in, The second auxiliary feature data is obtained by processing the first predetermined data using an auxiliary processing unit of the deep learning model, wherein the auxiliary processing unit includes a convolutional neural network.

25. The device according to any one of claims 19 to 23, in, The first auxiliary feature data includes first task feature data.

26. A training device for a deep learning model, include: A fourth obtaining module is used to obtain first intermediate sample feature data according to the sample image, wherein the first intermediate sample feature data includes first sample auxiliary feature data and first sample category feature data, and the first sample auxiliary feature data is data used to assist the multi-task recognition of the sample image; a fifth obtaining module, configured to obtain a sample task probability value and a sample category probability value of the sample image according to the first sample auxiliary feature data and the first sample category feature data, wherein the sample task probability value is obtained by processing the first sample auxiliary feature data using at least one auxiliary classification unit of the deep learning model, and the sample category probability value is obtained by processing the sample first category feature data using at least one category classification unit of the deep learning model, The sample task probability value includes the probability value of the sample image belonging to at least one task, and the sample category probability value includes the probability value of the sample image belonging to at least one category; and The sixth acquisition module is used to train the deep learning model using the sample task probability value, the sample category probability value and the label value of the sample image to obtain a trained deep learning model.

27. The device according to claim 26, in, The fourth acquisition module comprises: a third obtaining submodule, configured to obtain second intermediate sample feature data according to the sample image and predetermined sample data, wherein the second intermediate sample feature data includes second sample auxiliary feature data, second sample category feature data, and sample object feature data of each of a plurality of sample image blocks in the sample image; and The fourth obtaining submodule is used to extract at least one of the global feature and the local feature of the second intermediate sample feature data to obtain the first intermediate sample feature data.

28. The device according to claim 27, in, The predetermined sample data includes first predetermined sample data and second predetermined sample data; Wherein, the third obtaining submodule includes: a sixth obtaining unit, configured to process the sample image to obtain sample object feature data of each of a plurality of sample image blocks included in the sample image; a seventh obtaining unit, configured to process the first predetermined sample data to obtain the second sample auxiliary feature data; an eighth obtaining unit, configured to process the second predetermined sample data to obtain the second sample category characteristic data; and The ninth obtaining unit is used to obtain the second intermediate sample feature data according to the second sample auxiliary feature data, the second sample category feature data and the sample object feature data of each of the plurality of sample image blocks.

29. The device according to claim 28, in, The deep learning model includes an auxiliary processing unit; Wherein, the seventh obtaining unit comprises: The first obtaining subunit is used to process the first predetermined sample data using the auxiliary processing unit to obtain the second sample auxiliary feature data.

30. The device according to claim 27, in, The fourth obtaining submodule comprises: The tenth obtaining unit is used to process the second intermediate sample feature data based on the attention strategy and the local strategy to obtain the first intermediate sample feature data.

31. The device according to claim 30, in, The deep learning model includes a backbone module, the backbone module includes at least one cascaded backbone unit, and the backbone unit includes a multi-head self-attention layer and a feedforward network layer; Wherein, the tenth obtaining unit comprises: The second obtaining subunit is used to use the backbone module to process the second intermediate sample feature data to obtain the first intermediate sample feature data.

32. The device according to claim 31, in, The backbone module includes M cascaded backbone units, where M is an integer greater than or equal to 1; Wherein, the second obtaining subunit is used for: In the case of M>1 and m>1, Process the fourth intermediate sample feature data of the (m-1)th level using the multi-head self-attention layer of the mth level to obtain the third intermediate sample feature data of the mth level; Processing the third intermediate sample feature data of the mth level by using the feedforward network layer of the mth level to obtain fourth intermediate sample feature data of the mth level; and The first intermediate sample feature data is obtained according to the fourth intermediate sample feature data of the Nth level, wherein N is an integer greater than 1 and less than or equal to M.

33. The device according to claim 28, in, The sample object feature data is obtained by processing the sample image block using the object processing unit of the deep learning model; The second sample category feature data is obtained by processing the second predetermined sample data using the category processing unit of the deep learning model.

34. The device according to any one of claims 26 to 33, in, The sample task probability value is obtained by processing the first sample auxiliary feature data using the auxiliary classification unit of the deep learning model; Among them, the sample category probability value is obtained by processing the first sample category feature data using the category classification unit of the deep learning model.

35. The device according to any one of claims 26 to 33, in, The label value includes an auxiliary label value and a category label value; Wherein, the sixth acquisition module includes: A fifth obtaining submodule, configured to obtain a first output value based on a first loss function using the sample task probability value and the auxiliary label value; A sixth obtaining submodule, configured to obtain a second output value based on a second loss function using the sample category probability value and the category label value; a seventh obtaining submodule, configured to obtain the output value according to the first output value and the second output value; and An eighth acquisition submodule is used to adjust the model parameters of the deep learning model according to the output value until a predetermined end condition is met, thereby obtaining the trained deep learning model.

36. The device according to any one of claims 26 to 33, in, The first sample auxiliary feature data includes first sample task feature data.

37. An electronic device, include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute any one of claims 1 to 7 or any one of claims 8 to 18.

38. A non-transitory computer-readable storage medium storing computer instructions, in, The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 7 or any one of claims 8 to 18.

39. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7 or any one of claims 8 to 18.