Image processing method and device, electronic equipment and readable storage medium

Abstract

The embodiment of the invention provides an image processing method and device, electronic equipment and a readable storage medium, and relates to the technical field of computers. According to the embodiment of the invention, when the task networks of different predetermined tasks in the multi-task model are trained, the available training labels can be automatically screened out for training according to the task matching identifiers corresponding to the current predetermined tasks. Therefore, the data used by each task network during training can be the same as the data used by each task network during independent training, the training effect of the multi-task model is ensured, and the image processing accuracy of the multi-task model is improved. Meanwhile, the image samples do not need to be additionally labeled, so that the training efficiency of the multi-task model can be improved while the training effect of the multi-task model is ensured.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image processing method, device, electronic device and readable storage medium. Background Art

[0002] With the development of computer technology, it is now possible to integrate multiple predetermined tasks for image processing in the same electronic device to meet the needs of different scenarios. In related technologies, each predetermined task is often processed by an image processing model corresponding to the predetermined task. Therefore, as the number of predetermined tasks increases, the computing resources occupied by the predetermined tasks in the electronic device also increase.

[0003] In view of this situation, each task network in the multi-task model can be used to process each predetermined task (that is, the image processing models corresponding to multiple predetermined tasks are integrated into the same multi-task model). However, since each task network may be used to process different tasks, the training sets corresponding to each task network may be different, which will cause certain obstacles to the training process of the multi-task model, making the training efficiency and training effect of the multi-task model low, and thus making the accuracy of the multi-task model for image processing low. Therefore, how to effectively improve the performance of the multi-task model is an urgent problem to be solved. Summary of the invention

[0004] In view of this, the embodiments of the present application provide an image processing method, device, electronic device and readable storage medium, which improve the accuracy of image processing of the multi-task model and the training efficiency of the multi-task model.

[0005] In a first aspect, an image processing method is provided, the method comprising:

[0006] A target image is acquired, wherein the target image at least includes a face area.

[0007] Based on the task network corresponding to each predetermined task in the multi-task model, the target image is processed to determine the detection results corresponding to each predetermined task. The multi-task model is at least used for image processing tasks, and the multi-task model includes task networks corresponding to multiple predetermined tasks.

[0008] The multi-task model is trained by the following steps:

[0009] Obtain a comprehensive training set corresponding to the multi-task model, wherein the comprehensive training set includes at least image samples corresponding to each of the task networks, at least one training label corresponding to the image samples, and a task matching identifier corresponding to the training label, wherein the task matching identifier is used to identify the validity of each of the training labels for each of the predetermined tasks.

[0010] For each task network, a valid label corresponding to the task network and a valid sample corresponding to the valid label are determined in each training label according to the task matching identifier.

[0011] According to the valid samples and the valid labels, target losses corresponding to the task networks are determined.

[0012] The joint loss is determined according to the target losses respectively corresponding to the task networks.

[0013] Based on the joint loss, the multi-task model is trained.

[0014] In some embodiments, the predetermined task includes one or more combinations of the following tasks: a driver face detection task, a driver distraction state detection task, a driver fatigue state detection task, a driver dangerous action detection task, and a driver face occlusion detection task.

[0015] In some embodiments, determining the joint loss according to the target losses respectively corresponding to the task networks comprises:

[0016] Based on the target losses and preset weights respectively corresponding to the task networks, weighted summation is performed on the target losses to determine the joint loss, wherein the preset weights are determined based on at least the task type corresponding to the task network.

[0017] In some embodiments, the training labels include numerical labels.

[0018] Determining the target loss corresponding to each of the task networks according to the valid samples and the valid labels includes:

[0019] For each numerical label, the numerical label is normalized and encoded to determine a normalized numerical value corresponding to the numerical label.

[0020] According to the valid samples and the normalized values, target losses corresponding to the task networks are determined.

[0021] In some embodiments, the image sample includes a detected target area.

[0022] The step of normalizing and encoding the numerical label to determine the normalized numerical value corresponding to the numerical label includes:

[0023] According to the size of the detected target area, the predetermined adjustment parameter and the adjustment factor, the numerical label is normalized and encoded to determine the normalized numerical value corresponding to the numerical label.

[0024] In some embodiments, the method further comprises:

[0025] Get at least one raw sample.

[0026] According to the sample type corresponding to each of the original samples and the preset enhancement method corresponding to each of the sample types, data enhancement is performed on each of the original samples to determine an enhanced data set.

[0027] The comprehensive training set is generated based on each of the original samples and the enhanced data set.

[0028] In a second aspect, an image processing device is provided, the device comprising:

[0029] The target image acquisition module is configured to acquire a target image, wherein the target image at least includes a face area.

[0030] The detection result determination module is configured to execute a task network corresponding to each predetermined task in a multi-task model, process the target image, and determine the detection results corresponding to each predetermined task. The multi-task model is at least used for image processing tasks, and the multi-task model includes task networks corresponding to multiple predetermined tasks.

[0031] The multi-task model is trained through the following modules:

[0032] A comprehensive training set acquisition module is configured to execute acquisition of a comprehensive training set corresponding to a multi-task model, wherein the multi-task model is at least used for image processing tasks, and the multi-task model includes task networks corresponding to multiple predetermined tasks respectively. The comprehensive training set includes at least image samples corresponding to each of the task networks, at least one training label corresponding to the image samples, and a task matching identifier corresponding to the training label, wherein the task matching identifier is used to identify the validity of each of the training labels for each of the predetermined tasks.

[0033] The training data screening module is configured to perform, for each task network, determining, in each of the training labels, a valid label corresponding to the task network and a valid sample corresponding to the valid label according to the task matching identifier.

[0034] The target loss determination module is configured to determine the target loss corresponding to each of the task networks according to the valid samples and the valid labels.

[0035] The joint loss determination module is configured to determine the joint loss according to the target losses respectively corresponding to the task networks.

[0036] The training module is configured to perform training of the multi-task model based on the joint loss.

[0037] In some embodiments, the predetermined task includes one or more combinations of the following tasks: a driver face detection task, a driver distraction state detection task, a driver fatigue state detection task, a driver dangerous action detection task, and a driver face occlusion detection task.

[0038] In some embodiments, the joint loss determination module is specifically configured to perform:

[0039] Based on the target losses and preset weights respectively corresponding to the task networks, weighted summation is performed on the target losses to determine the joint loss, wherein the preset weights are determined based on at least the task type corresponding to the task network.

[0040] In some embodiments, the training labels include numerical labels.

[0041] The target loss determination module is specifically configured to execute:

[0042] For each numerical label, the numerical label is normalized and encoded to determine a normalized numerical value corresponding to the numerical label.

[0043] According to the valid samples and the normalized values, target losses corresponding to the task networks are determined.

[0044] In some embodiments, the image sample includes a detected target area.

[0045] The target loss determination module is specifically configured to execute:

[0046] According to the size of the detected target area, the predetermined adjustment parameter and the adjustment factor, the numerical label is normalized and encoded to determine the normalized numerical value corresponding to the numerical label.

[0047] In some embodiments, the apparatus further comprises:

[0048] The original sample acquisition module is configured to execute acquisition of at least one original sample.

[0049] The data enhancement module is configured to perform data enhancement on each of the original samples according to the sample type corresponding to each of the original samples and the preset enhancement method corresponding to each of the sample types, and determine an enhanced data set.

[0050] The training set generation module is configured to generate the comprehensive training set based on the original samples and the enhanced data set.

[0051] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method described in the first aspect.

[0052] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described in the first aspect is implemented.

[0053] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the method described in the first aspect.

[0054] When training the task network for different scheduled tasks in the multi-task model, the embodiment of the present application can automatically filter out the available training tags for training according to the task matching identifier corresponding to the current scheduled task. Therefore, the above method can make the data used by each task network during training the same as the data used when training separately, thereby ensuring the training effect of the multi-task model and improving the accuracy of the multi-task model for image processing. At the same time, since the embodiment of the present application does not need to perform additional annotations on the image samples, the embodiment of the present application can improve the training efficiency of the multi-task model while ensuring the training effect of the multi-task model. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The above and other purposes, features and advantages of the embodiments of the present application will become more apparent through the following description of the embodiments of the present application with reference to the accompanying drawings, in which:

[0056] Figure 1 A flowchart of an image processing method according to an embodiment of the present application;

[0057] Figure 2 A schematic diagram of the structure of a multi-task model according to an embodiment of the present application;

[0058] Figure 3 This is a flow chart of a training method for a multi-task model according to an embodiment of the present application;

[0059] Figure 4 A flowchart of normalization processing for an embodiment of the present application;

[0060] Figure 5 A schematic diagram of a detected target area according to an embodiment of the present application;

[0061] Figure 6 A schematic diagram of test results obtained by testing a task network corresponding to a plurality of predetermined tasks according to an embodiment of the present application;

[0062] Figure 7 A flowchart for generating a comprehensive training set for an embodiment of the present application;

[0063] Figure 8 A schematic diagram of the structure of an image processing device according to an embodiment of the present application;

[0064] Fig. 9 A schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0065] The present application is described below based on embodiments, but the present application is not limited to these embodiments. In the detailed description of the present application below, some specific details are described in detail. It is possible for those skilled in the art to fully understand the present application without the description of these details. In order to avoid confusing the essence of the present application, known methods, processes, flows, components and circuits are not described in detail.

[0066] In addition, persons of ordinary skill in the art will appreciate that the drawings provided herein are for illustration purposes and are not necessarily drawn to scale.

[0067] Unless the context clearly requires otherwise, the words "include", "comprising" and the like throughout this application should be interpreted as including rather than exclusive or exhaustive; that is, as meaning "including but not limited to".

[0068] In the description of this application, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of this application, unless otherwise specified, the meaning of "plurality" is two or more.

[0069] The solutions described in this specification and in the examples, if they involve the processing of personal information, will be processed on the premise of having a legal basis (such as obtaining the consent of the subject of personal information, or being necessary for the performance of a contract, etc.), and will only be processed within the scope of regulations or agreements. If a user refuses to process personal information other than the necessary information for basic functions, it will not affect the user's use of basic functions.

[0070] With the development of computer technology, it is now possible to integrate multiple predetermined tasks for image processing in the same electronic device to meet the needs of different scenarios. Taking online car-hailing services as an example, in order to ensure the driving safety of drivers (i.e. online car-hailing drivers), online car-hailing service platforms can collect driver images through online car-hailing dedicated camera equipment after authorization, and the online car-hailing dedicated camera equipment will perform image processing such as driver distraction state detection, driver fatigue state detection, driver dangerous action detection, and driver face occlusion detection based on the driver image to determine whether the driver is in a safe driving state. Among them, the relevant technology will set up a separate image processing model for each detection task, and each image processing model needs to perform feature extraction and feature processing operations according to task requirements.

[0071] For example, when the related technology processes the driver image through the image processing model of driver distraction state detection, it will extract and process the features such as eyes and head rotation. When the driver image is processed through the image processing model of driver fatigue state detection, it will extract and process the features such as eyes and mouth. When the driver image is processed through the image processing model of driver dangerous action detection, it will extract and process the features such as mouth and ear area.

[0072] However, since each image processing model needs to occupy a certain amount of computing resources, the related technology often has problems such as waste of computing resources and excessive use of computing resources during image processing. For example, in the scenario of the above-mentioned online car-hailing service, the image processing task generally needs to be independently performed by the camera equipment dedicated to the above-mentioned online car-hailing. However, due to the influence of the hardware resources of the camera equipment itself, the computing resources of the camera equipment are often limited, and it is impossible to perform too many image processing tasks at the same time. In other words, when there are too many image processing tasks to be processed (that is, when there are too many image processing models to be run), the camera equipment will be under greater computing pressure. In other words, in the related technology, each scheduled task is often processed by the image processing model corresponding to the scheduled task. Therefore, as the number of scheduled tasks increases, the computing resources occupied by the scheduled tasks in the electronic device also increase accordingly.

[0073] In view of this situation, each task network in the multi-task model can be used to process each predetermined task (that is, the image processing models corresponding to multiple predetermined tasks are integrated into the same multi-task model). However, since each task network may be used to process different tasks respectively, there may be certain differences in the training sets corresponding to each task network, which will cause certain obstacles to the training process of the multi-task model. The usual practice of the related art is to perform label completion processing on the training sets corresponding to different predetermined tasks so that each training set includes the training labels corresponding to each predetermined task. Further, the related art will merge the training sets after the completion processing into a total training set, and train the multi-task model through the total training set. However, in the above process, the label completion processing will consume a lot of time, and the image samples after the label completion processing are often negative samples, which will cause an imbalance in the ratio of positive and negative samples, making the training effect and training efficiency of the multi-task model low, and then make the accuracy of the multi-task model for image processing low.

[0074] Therefore, how to improve the training efficiency of the multi-task model while ensuring the training effect of the multi-task model is an urgent problem that needs to be solved.

[0075] In order to solve the above problems, the embodiment of the present application provides an image processing method to ensure the training effect of the multi-task model while improving the training efficiency of the multi-task model, thereby improving the accuracy of the multi-task model for image processing. Among them, the image processing method can be applied to electronic devices, the electronic device can be a terminal or a server, the terminal can be a smart phone, a tablet computer, a vehicle control terminal, a camera device (such as a driving recorder or a camera device dedicated to online car-hailing) or a personal computer (Personal Computer, PC), etc., and the server can be a single server, or a server cluster configured in a distributed manner, or a cloud server.

[0076] Specifically, Figure 1 As shown, the image processing method of the embodiment of the present application may include the following steps:

[0077] In step S110, a target image is acquired.

[0078] Among them, the target image at least includes a face area. The target image can be directly captured by a camera unit set in the electronic device itself (such as a camera of a smartphone, a camera of a driving recorder, or a camera of a camera device dedicated to an online car-hailing service), or it can be captured by a camera device external to the electronic device (wired connection or wireless connection).

[0079] In step S120, based on the task network corresponding to each predetermined task in the multi-task model, the target image is processed to determine the detection results corresponding to each predetermined task.

[0080] Among them, the multi-task model of the embodiment of the present application is at least used for image processing tasks. The multi-task model may include a task network corresponding to multiple predetermined tasks. That is to say, each predetermined task of the embodiment of the present application may be a subtask under the image processing task.

[0081] In an optional embodiment, the multi-task model may include a main extraction network for one-time feature extraction, and then determine the shared feature map through the main extraction network, and the task network corresponding to each predetermined task extracts and selects the features it needs for processing.

[0082] For example, Figure 2 As shown, the multi-task model of the embodiment of the present application may include a target detection network 21, a main extraction network 22 and multiple task networks (task network 231-task network 23n, n is a natural number greater than or equal to 1).

[0083] Among them, the target detection network 21 can be used to perform target detection tasks (such as face detection, vehicle detection and other tasks) to determine the target detection results (i.e., the detected target area). The main extraction network 22 can be used to determine the shared feature map of the detected target area, which can include the image features required for each predetermined task (i.e., task network 231-task network 23n), and the predetermined task can be each subtask under a total image processing task. For example, the total image processing task can be a driving safety detection task, and the predetermined task can be a driver face detection task, a driver distraction state detection task, a driver fatigue state detection task, a driver dangerous action detection task, and a driver face occlusion detection task and other subtasks under the driving safety detection task. For another example, the total image processing task can also be an illegally parked vehicle detection task, and the predetermined task can be a vehicle detection task, a parking space detection task, and a vehicle state detection task and other subtasks under the illegally parked vehicle detection task.

[0084] Based on the multi-task model, the embodiment of the present application can determine the shared feature map by a one-time feature extraction method, and the task network corresponding to each predetermined task can extract and select the features required by itself for processing. Therefore, compared with the conventional method of setting up a detection model separately, the embodiment of the present application can effectively reduce the number of feature extractions (i.e., from n feature extractions to 1 feature extraction), and under the premise of meeting the accuracy requirements, reduce the computing pressure of electronic equipment (especially equipment with less computing resources such as camera equipment dedicated to online car-hailing), and achieve the purpose of simply and efficiently executing various image processing tasks.

[0085] Further, such as Figure 3 As shown, the multi-task model of the embodiment of the present application can be trained by the following steps:

[0086] In step S210, a comprehensive training set corresponding to the multi-task model is obtained.

[0087] The comprehensive training set includes at least image samples corresponding to each task network, at least one training label corresponding to the image sample, and a task matching identifier corresponding to the training label, and the task matching identifier is used to identify the validity of each training label for each predetermined task.

[0088] In an optional embodiment, the predetermined tasks of the embodiment of the present application include one or more combinations of the following tasks: driver face detection task, driver distraction state detection task, driver fatigue state detection task, driver dangerous action detection task and driver face occlusion detection task.

[0089] Through comprehensive detection of driver face detection tasks, driver distraction status detection tasks, driver fatigue status detection tasks, driver dangerous action detection tasks and driver face occlusion detection tasks, the embodiments of the present application can effectively determine whether the driver is in a safe driving state, thereby improving the driver's driving safety.

[0090] Furthermore, for the training process of the multi-task model, the task matching identifier can be used to identify the validity of each training label for each predetermined task. For example, the embodiment of the present application can set the task matching identifier corresponding to the unusable training label (i.e., invalid training label) to invalid for each predetermined task. If the unusable training label is a training label corresponding to a classification task, the embodiment of the present application can set the task matching identifier corresponding to the training label to "-1"; if the unusable training label is a training label corresponding to a regression task, the embodiment of the present application can set the task matching identifier corresponding to the training label to "-inf" (i.e., negative infinity).

[0091] For example, there are scheduled tasks A1, B1, and C1, and scheduled tasks A1, B1, and C1 correspond to training sets A2, B2, and C2 respectively, and the training labels in training sets A2, B2, and C2 cannot be used to train the task networks of scheduled tasks corresponding to other training sets. In other words, the training labels in training set A2 cannot be used to train the task networks of scheduled tasks B1 and C1, the training labels in training set B2 cannot be used to train the task networks of scheduled tasks A1 and C1, and the training labels in training set C2 cannot be used to train the task networks of scheduled tasks A1 and B1.

[0092] Furthermore, the embodiment of the present application may set a task matching identifier for each training label according to the validity of each training label for each predetermined task. Specifically, for the predetermined task A1, the embodiment of the present application may set the task matching identifier corresponding to the training labels in the training sets B2 and C2 to invalid ("-1" or "-inf"). For the predetermined task B1, the embodiment of the present application may set the task matching identifier corresponding to the training labels in the training sets A2 and C2 to invalid ("-1" or "-inf"). For the predetermined task C1, the embodiment of the present application may set the task matching identifier corresponding to the training labels in the training sets A2 and B2 to invalid ("-1" or "-inf").

[0093] That is to say, the same training label may have different task matching identifiers for different scheduled tasks. When the embodiment of the present application is training the task network for different scheduled tasks, it can automatically filter out the available training labels for training according to the task matching identifier corresponding to the current scheduled task. Therefore, through the above method, the data used by each task network during training can be the same as the data used when training separately, thereby ensuring the training effect of the multi-task model. At the same time, since the embodiment of the present application does not require additional annotation of the image samples, the embodiment of the present application can improve the training efficiency of the multi-task model while ensuring the training effect of the multi-task model.

[0094] It should be noted that the above example is only one way to set the task matching identifier in the embodiment of the present application, and it does not limit the specific method of setting the task matching identifier. For example, the embodiment of the present application can also set a task matching identifier that only includes a valid identifier (such as "1"). Further, the embodiment of the present application can filter each training label by the valid identifier. In another case, the embodiment of the present application can also set a task matching identifier that includes both a valid identifier (such as "1") and an invalid identifier (such as "-1" or "-inf").

[0095] In step S220, for each task network, a valid label corresponding to the task network and a valid sample corresponding to the valid label are determined in each training label according to the task matching identifier.

[0096] Specifically, the embodiment of the present application can filter out the unmatched training labels (e.g., training labels with task matching identifiers of "-1" or "-inf") in each training label according to the task matching identifier to determine the valid labels and the valid samples corresponding to the valid labels.

[0097] In step S230, the target loss corresponding to each task network is determined according to the valid samples and the valid labels.

[0098] Specifically, the embodiment of the present application can input valid samples into the task network to determine the output result of the task network, and determine the target loss corresponding to the task network according to the output result and the valid label.

[0099] In an optional embodiment, the training labels of the embodiment of the present application may include numerical labels. Among them, the numerical labels can be used to train the task network built based on the regression model, and the target loss can be an absolute loss function (Smooth L1Loss). The absolute loss function has a high correlation with the training label (that is, the effective label used in the embodiment of the present application during training). The relevant technology generally uses an absolute value (such as the actual pixel distance in the image) or a relative value (such as the value obtained by scaling the absolute value according to the ratio of the image) as the training label.

[0100] Taking eyelid distance detection as an example, the comprehensive training set includes sample image X1 (image size is 320 pixels * 320 pixels, eyelid distance in the figure is 16 pixels) and sample image X2 (image size is 160 pixels * 160 pixels, eyelid distance in the figure is 16 pixels). During the training process, it is often necessary to resize each sample image to the same size. Taking 160 pixels * 160 pixels as an example, the image size of sample image X1 after resizing is 160 pixels * 160 pixels, and the eyelid distance in the figure is 8 pixels. The size of sample image X2 does not need to be resized.

[0101] In one case, if the absolute value is used as the training label, the training labels corresponding to the sample image X1 and the sample image X2 are both 16 pixels. However, since the sample image X1 will be resized during training, the training label corresponding to the sample image X1 is too large relative to the resized sample image X1, and the training label at this time is no longer applicable to the resized sample image X1. In other words, if the absolute value is used as the training label, it will affect the training effect to a certain extent.

[0102] In another case, if relative values ​​are used as training labels, the training label corresponding to sample image X1 can be 16 pixels / 320 pixels = 0.05, and the training label corresponding to sample image X2 can be 16 pixels / 160 pixels = 0.1. By processing relative values, the problems caused by using absolute values ​​as training labels can be avoided. However, it can be seen from the above process that using relative values ​​as training labels will make the pixels of the training labels too small, and too small training labels will affect the convergence of the model to a certain extent, thereby affecting the training effect.

[0103] In order to avoid the problems caused by using absolute values ​​or relative values ​​as training labels, the embodiment of the present application can adopt a normalized encoding method to normalize the training labels to determine normalized values, and determine the target loss based on the normalized values.

[0104] Specifically, Figure 4 As shown, the above step S230 may specifically include the following steps:

[0105] In step S231, for each numerical label, the numerical label is normalized and encoded to determine the normalized numerical value corresponding to the numerical label.

[0106] Specifically, the embodiment of the present application can set an adjustment factor, and according to the range determined by the adjustment factor, the normalized value determined by the normalized code is mapped to a reasonable value range, thereby avoiding the problems caused by the unreasonable training label values ​​mentioned above and improving the training effect.

[0107] In an optional implementation manner, the image sample of the embodiment of the present application includes a detected target area, wherein the detected target is a face area.

[0108] For example, Figure 5 As shown, Figure 5 This is a scene of driver driving safety detection. Image 51 can be directly captured by a camera unit provided in the electronic device itself (such as a camera of a smartphone, a camera of a driving recorder, or a camera of a camera device dedicated to an online car-hailing service), or it can be captured by a camera device external to the electronic device (wired connection or wireless connection). After determining image 51, the embodiment of the present application can perform face detection on image 51 to determine a face area image 52 of image 51, wherein the face area image 52 is the target area to be detected.

[0109] Furthermore, the above step S231 may be executed as follows: performing normalized encoding on the numerical labels according to the size of the detected target area, the predetermined adjustment parameters and the adjustment factors, and determining the normalized numerical values ​​corresponding to the numerical labels.

[0110] Among them, the adjustment parameter can be a predetermined value or a statistical percentile value obtained based on historical data statistics. The adjustment factor is a preset value. The embodiment of the present application can determine a numerical range based on the adjustment factor. For example, if the adjustment factor is S, the numerical range determined by the embodiment of the present application can be [-S, +S]. In this way, the normalized numerical value finally determined by the embodiment of the present application will be mapped between the adjustment factor -S and +S, so that each numerical label is mapped to a reasonable numerical value.

[0111] For example, taking the offset as an example, the offset can be used to characterize the displacement of the detected target in the image (usually the offset distance is expressed in pixels), which is often used in the target tracking process. During the training process, the comprehensive training set of the embodiment of the present application may include at least image samples, sample target detection frames of the image samples, and random transformation detection frames corresponding to the offset. Among them, the sample target detection frame is used to identify the actual area of ​​the detected target in the image sample (that is, the offset corresponding to the sample target detection frame is 0), and the random transformation detection frame corresponding to the offset is the detection frame obtained by the sample target detection frame after the random offset transformation.

[0112] Since the numerical label corresponding to the offset is usually a randomly determined value, the embodiment of the present application can determine the adjustment parameter in a predetermined manner to avoid the value of the randomly determined offset being too large. Specifically, the process of normalizing the offset can be characterized by the following formula:

[0113] l norm =S*l abs / (h*0.8)

[0114] Among them, l norm It is used to characterize the normalized value. S is the adjustment factor. The normalized value finally determined in the embodiment of the present application will be mapped between the adjustment factors -S and +S. abs is the size of the numerical label (in this case, the offset). h is the height of the detected target area. 0.8 is a predetermined adjustment parameter. The adjustment parameter is a parameter used to limit the maximum offset, which can avoid excessive offset of the random transformation detection frame. For example, if the sample target detection frame is an area of ​​200 pixels * 200 pixels (i.e., h is 200 pixels), the maximum offset allowed by the embodiment of the application (i.e., l abs The maximum value of () is 200 pixels * 0.8 = 160 pixels. That is to say, if the starting point of the horizontal coordinate of the sample target detection frame in the image is 726 pixels and the end point is 926 pixels (726 pixels + 200 pixels), then through the above adjustment parameter (0.8), the embodiment of the present application can determine that the starting point of the horizontal coordinate of the random transformation detection frame can be any random value from 566 pixels (726 pixels - 160 pixels) to 886 pixels (726 pixels + 160 pixels).

[0115] For another example, taking the zoom amount as an example, the zoom amount can be used to characterize the magnified / reduced size of the detected target in the image (usually the zoom size is expressed in pixels), which is often used in the target tracking process. During the training process, the comprehensive training set of the embodiment of the present application may include at least image samples, sample target detection frames of the image samples, and random transformation detection frames corresponding to the zoom amount. Among them, the sample target detection frame is used to identify the actual area of ​​the detected target in the image sample (that is, the zoom amount corresponding to the sample target detection frame is 0), and the random transformation detection frame corresponding to the offset corresponding to the zoom amount is the detection frame obtained by the sample target detection frame after random scaling transformation.

[0116] Since the numerical label corresponding to the scaling amount is usually a randomly determined value, the embodiment of the present application can determine the adjustment parameter in a predetermined manner to avoid the randomly determined scaling amount being too large. Specifically, the process of normalizing the scaling amount can be represented by the following formula:

[0117] l norm =S*l abs / ((h-2*l abs )*0.15))

[0118] Among them, l abs is the size of the numerical label (in this case, the scaling amount). 0.15 is a predetermined adjustment parameter. Similar to the above formula, 0.15 in this formula is a parameter used to limit the maximum scaling amount, which can avoid excessive scaling of the random transformation detection frame. For example, if the sample target detection frame is an area of ​​200 pixels * 200 pixels, then through the above adjustment parameter (0.15), the embodiment of the present application can determine the size of the random transformation detection frame to be any size from 170 pixels * 170 pixels to 230 pixels * 230 pixels.

[0119] For another example, taking the eyelid distance as an example, the eyelid distance can be used to characterize the distance between the eyelids of both eyes. During the training process, the comprehensive training set of the embodiment of the present application may at least include image samples and numerical labels (eyelid distances) corresponding to the image samples.

[0120] Since the numerical label corresponding to the eyelid distance is usually a numerical value determined based on the actual image, the embodiment of the present application can determine a statistical percentile value as an adjustment parameter by statistically analyzing historical data. Specifically, the process of normalizing and encoding the eyelid distance can be represented by the following formula:

[0121] l norm =S*l abs / (h*0.05)

[0122] Among them, l absis the size of the numerical label (in this case, the eyelid distance). 0.05 is the statistical quantile value (i.e., the 995th quantile value). The quantile value is one of the characteristic numbers of a random variable, which serves to cut off excessively large values ​​in the embodiment of the present application. The embodiment of the present application can determine the statistical quantile value by statistically analyzing historical data.

[0123] In step S232, the target loss corresponding to each task network is determined according to the valid samples and the normalized values.

[0124] Specifically, the embodiment of the present application can input valid samples into the task network to determine the output result of the task network, and determine the target loss corresponding to the task network based on the output result and the normalized value.

[0125] It should be noted that if the embodiment of the present application determines the target loss corresponding to the task network based on the normalized numerical value, and trains the multi-task model based on the target loss, the multi-task model will perform normalized decoding processing accordingly during the reasoning process.

[0126] For example, if normalized decoding is performed on the offset, the normalized decoding process can be represented by the following formula:

[0127] l abs = l pred *h*0.8 / S

[0128] Among them, l abs is the actual offset corresponding to the detected target area. pred is the intermediate result output by the multi-task model, that is, the predicted value output by the multi-task model. h is the height of the detected target area. S is the adjustment factor, and 0.8 is a predetermined adjustment parameter.

[0129] In addition, if normalized decoding is performed on the eyelid distance, the embodiment of the present application can perform normalized decoding based on a method similar to the formula (ie, replacing the predetermined adjustment parameter with an adjustment parameter in the form of a statistical percentile value).

[0130] If normalized decoding is performed for the scaling amount, the normalized decoding process can be expressed as the following formula:

[0131]

[0132] l abs =h*l mid / (1+2 / l mid )

[0133] Among them, l abs is the actual scaling value corresponding to the detected target area. mid For the embodiment of this application, predThe intermediate conversion result is determined, and 0.15 is a predetermined adjustment parameter.

[0134] The embodiment of the present application determines the normalized value so that the training labels of all image samples are mapped to a reasonable value range (i.e., the range determined by the above adjustment factor), thereby avoiding the problems caused by the unreasonable values ​​of the above training labels and improving the training effect. Furthermore, in the process of model reasoning, the embodiment of the present application can reversely decode the actual value through normalized decoding, thereby improving the accuracy of the offset parameter.

[0135] In step S240, a joint loss is determined according to the target losses corresponding to each task network.

[0136] Among them, the joint loss is the loss used to adjust the overall model parameters of the multi-task model. The embodiment of the present application can perform weighted sum processing on the target losses corresponding to each task network to determine the joint loss.

[0137] In this process, since the data and task objectives used by each predetermined task are often different, there may be a "negative transfer phenomenon" when the task networks of each predetermined task are jointly trained, that is, the learning of one task will interfere with or even inhibit the learning of another task. Therefore, how to make all task networks converge together during the joint training process is one of the difficulties of joint training.

[0138] To address this issue, in an optional implementation, the above step S240 may be performed as follows: based on the target losses and preset weights corresponding to the task networks, weighted summation of the target losses is performed to determine the joint loss.

[0139] The preset weight is determined based on at least the task type corresponding to the task network. In practical applications, the embodiment of the present application can determine the task type corresponding to the task network according to the features concerned by each task network, the fluctuation range of the training effect of each task network affected by the weight (i.e., the difficulty of training), and the number of image samples of each task network.

[0140] For example, Figure 6 As shown, Figure 6 This is a schematic diagram of the test results obtained by testing the task networks corresponding to multiple predetermined tasks in the embodiment of the present application. The curves in the test results are used to characterize the training effects of each task network under different weights. Figure 6 The curve in is used to characterize the parameter corresponding to the training effect, for example, the parameter may be the F1 score (F1-Score) or the R-square (R-square). Figure 6The corresponding predetermined tasks include a lowered eyes detection task (i.e., detecting whether the driver is looking downward), a closed eyes detection task, a glasses wearing detection task, an eyelid distance detection task, a mask wearing detection task, and a yawn detection task.

[0141] Specifically, test result 61 is the test result of the task network corresponding to the low eye detection task, test result 62 is the test result of the task network corresponding to the closed eyes detection task, test result 63 is the test result of the task network corresponding to the glasses wearing detection task, test result 64 is the test result of the task network corresponding to the eyelid distance detection task, test result 65 is the test result of the task network corresponding to the mask wearing detection task, and test result 66 is the test result of the task network corresponding to the yawn detection task.

[0142] Depend on Figure 6 It can be seen that test results 61, 62, 63 and 64 are all eye-related detection tasks, and the fluctuation indexes of the training effects corresponding to these four tasks are relatively small, which is specifically manifested in that the training process is relatively stable and the overall trend is relatively close. Test results 65 and 66 are all mouth-related detection tasks, and the fluctuation indexes of the training effects corresponding to these two tasks are relatively large. Figure 6 According to the test results shown, the embodiment of the present application can determine the training difficulty corresponding to each predetermined task, that is, the predetermined task with a relatively stable training process is a task with low difficulty, and the predetermined task with a large fluctuation in the training process is a task with high difficulty.

[0143] In addition, the embodiment of the present application can also determine the dominant task and the inferior task in each predetermined task according to the number of image samples of each predetermined task. Specifically, the predetermined task with a large number of image samples is the dominant task, which has a faster convergence speed and is easy to affect the training of other predetermined tasks. Correspondingly, the predetermined task with a small number of image samples is the inferior task, which has a slower convergence speed, and the training process is easily affected by other predetermined tasks.

[0144] Further, for disadvantaged tasks with low difficulty, the embodiment of the present application can appropriately reduce the weight corresponding to the task to reduce its convergence speed. For advantageous tasks with high difficulty, the embodiment of the present application can appropriately increase the weight corresponding to the task to speed up its convergence speed. For advantageous tasks with low difficulty, the embodiment of the present application can set a more moderate weight. For disadvantaged tasks with high difficulty, the embodiment of the present application can appropriately supplement the corresponding training data to ensure its training effect.

[0145] Therefore, the embodiment of the present application determines the preset weights corresponding to each task network by the task type corresponding to the task network, which can effectively balance the weights of each task network during the training process to avoid the above-mentioned "negative transfer phenomenon" and make all task networks converge together during the joint training process, thereby improving the joint training effect for the multi-task model.

[0146] In step S250, a multi-task model is trained based on the joint loss.

[0147] Specifically, the embodiments of the present application can perform backpropagation on the multi-task model based on the joint loss to adjust the overall model parameters of the multi-task model and realize the training of the multi-task model.

[0148] In this process, when the embodiment of the present application is training the task network for different scheduled tasks in the multi-task model, the available training labels can be automatically screened out for training according to the task matching identifier corresponding to the current scheduled task. Therefore, the above method can make the data used by each task network during training the same as the data used when training separately, thereby ensuring the training effect of the multi-task model and improving the accuracy of the multi-task model for image processing. At the same time, since the embodiment of the present application does not need to perform additional annotations on the image samples, the embodiment of the present application can improve the training efficiency of the multi-task model while ensuring the training effect of the multi-task model.

[0149] In an optional implementation, the embodiment of the present application can also generate a comprehensive training set by data enhancement, specifically, Figure 7 As shown, the embodiment of the present application may also include the following steps:

[0150] In step S310, at least one original sample is acquired.

[0151] Among them, the embodiments of the present application can directly obtain image data as original samples through legal channels.

[0152] In step S320, data enhancement is performed on each original sample according to the sample type corresponding to each original sample and the preset enhancement mode corresponding to each sample type, and an enhanced data set is determined.

[0153] Among them, since the multi-task model of the embodiment of the present application brings together multiple task networks, and there are certain differences in the type of data required by each task network, if a unified method is used to enhance the data of the original samples, it may have an adverse effect on the training of the task network.

[0154] Therefore, the embodiment of the present application can perform data enhancement on each original sample according to the sample type corresponding to each original sample and the preset enhancement method corresponding to each sample type, and determine the enhanced data set. Specifically, the sample type and the preset enhancement method corresponding to each sample type can be shown in the following Table 1:

[0155] Table 1

[0156]

[0157] According to the corresponding relationship shown in Table 1, the embodiment of the present application can perform corresponding data enhancement on each original sample through different enhancement methods, determine the enhanced data set, and improve the adaptability of the image samples.

[0158] In step S330, a comprehensive training set is generated based on the original samples and the enhanced data set.

[0159] The embodiment of the present application can perform targeted data enhancement on the original samples through the sample types corresponding to each original sample and the preset enhancement methods corresponding to each sample type, so that each image sample in the comprehensive training set can effectively participate in the training, thereby improving the training effect.

[0160] Based on the same technical concept, the embodiment of the present application also provides an image processing device, such as Figure 8 As shown, the device includes: a target image acquisition module 81, a detection result determination module 82, a comprehensive training set acquisition module 83, a training data screening module 84, a target loss determination module 85, a joint loss determination module 86 and a training module 87.

[0161] The target image acquisition module 81 is configured to acquire a target image, wherein the target image at least includes a face area.

[0162] The detection result determination module 82 is configured to execute a task network corresponding to each predetermined task in the multi-task model, process the target image, and determine the detection results corresponding to each predetermined task. The multi-task model is at least used for image processing tasks, and the multi-task model includes task networks corresponding to multiple predetermined tasks.

[0163] The multi-task model is trained through the following modules:

[0164] The comprehensive training set acquisition module 83 is configured to execute the acquisition of a comprehensive training set corresponding to a multi-task model, wherein the multi-task model is at least used for image processing tasks, and the multi-task model includes task networks corresponding to multiple predetermined tasks respectively. The comprehensive training set includes at least image samples corresponding to each of the task networks, at least one training label corresponding to the image samples, and a task matching identifier corresponding to the training label, and the task matching identifier is used to identify the validity of each of the training labels for each of the predetermined tasks.

[0165] The training data screening module 84 is configured to perform, for each task network, determining, in each of the training labels, a valid label corresponding to the task network and a valid sample corresponding to the valid label according to the task matching identifier.

[0166] The target loss determination module 85 is configured to determine the target loss corresponding to each of the task networks according to the valid samples and the valid labels.

[0167] The joint loss determination module 86 is configured to determine the joint loss according to the target losses respectively corresponding to the task networks.

[0168] The training module 87 is configured to perform training of the multi-task model based on the joint loss.

[0169] In some embodiments, the predetermined task includes one or more combinations of the following tasks: a driver face detection task, a driver distraction state detection task, a driver fatigue state detection task, a driver dangerous action detection task, and a driver face occlusion detection task.

[0170] In some embodiments, the joint loss determination module 86 is specifically configured to perform:

[0171] Based on the target losses and preset weights respectively corresponding to the task networks, weighted summation is performed on the target losses to determine the joint loss, wherein the preset weights are determined based on at least the task type corresponding to the task network.

[0172] In some embodiments, the training labels include numerical labels.

[0173] The target loss determination module 85 is specifically configured to execute:

[0174] For each numerical label, the numerical label is normalized and encoded to determine a normalized numerical value corresponding to the numerical label.

[0175] According to the valid samples and the normalized values, target losses corresponding to the task networks are determined.

[0176] In some embodiments, the image sample includes a detected target area.

[0177] The target loss determination module 85 is specifically configured to execute:

[0178] According to the size of the detected target area, the predetermined adjustment parameter and the adjustment factor, the numerical label is normalized and encoded to determine the normalized numerical value corresponding to the numerical label.

[0179] In some embodiments, the apparatus further comprises:

[0180] The original sample acquisition module is configured to execute acquisition of at least one original sample.

[0181] The data enhancement module is configured to perform data enhancement on each of the original samples according to the sample type corresponding to each of the original samples and the preset enhancement method corresponding to each of the sample types, and determine an enhanced data set.

[0182] The training set generation module is configured to generate the comprehensive training set based on the original samples and the enhanced data set.

[0183] When training the task network for different scheduled tasks in the multi-task model, the embodiment of the present application can automatically filter out the available training tags for training according to the task matching identifier corresponding to the current scheduled task. Therefore, the above method can make the data used by each task network during training the same as the data used when training separately, thereby ensuring the training effect of the multi-task model and improving the accuracy of the multi-task model for image processing. At the same time, since the embodiment of the present application does not need to perform additional annotations on the image samples, the embodiment of the present application can improve the training efficiency of the multi-task model while ensuring the training effect of the multi-task model.

[0184] Fig. 9 Schematic diagram of an electronic device according to an embodiment of the present application. Fig. 9 As shown, Fig. 9The electronic device shown is a general address query device, which includes a general computer hardware structure, which at least includes a processor 91 and a memory 92. The processor 91 and the memory 92 are connected via a bus 93. The memory 92 is suitable for storing instructions or programs executable by the processor 91. The processor 91 can be an independent microprocessor or a collection of one or more microprocessors. Thus, the processor 91 executes the instructions stored in the memory 92, thereby executing the method flow of the embodiment of the present application as described above to realize the processing of data and the control of other devices. The bus 93 connects the above-mentioned multiple components together, and at the same time connects the above-mentioned components to the display controller 94 and the display device and the input / output (I / O) device 95. The input / output (I / O) device 95 can be a mouse, a keyboard, a modem, a network interface, a touch input device, a somatosensory input device, a printer, and other devices known in the art. Typically, the input / output device 95 is connected to the system via an input / output (I / O) controller 96.

[0185] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices (equipment) or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application may adopt a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0186] The present application is described with reference to flowcharts of methods, apparatuses (devices) and computer program products according to embodiments of the present application. It should be understood that each process in the flowchart can be implemented by computer program instructions.

[0187] These computer program instructions may be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device that implements the process Figure 1 A function specified in a process or multiple processes.

[0188] These computer program instructions may also be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce the instructions for implementing the process Figure 1 A device that specifies functions in a process or multiple processes.

[0189] Another embodiment of the present application relates to a non-volatile storage medium for storing a computer-readable program, wherein the computer-readable program is used for a computer to execute part or all of the above method embodiments.

[0190] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by specifying relevant hardware through a program, and the program is stored in a storage medium, including several instructions for a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0191] Another embodiment of the present application relates to a computer program product, including a computer program / instruction, which can implement part or all of the above method embodiments when the computer program / instruction is executed by a processor.

[0192] That is, those skilled in the art can understand that the embodiments of the present application can specify relevant hardware (including the processor itself) through the processor executing a computer program product (computer program / instructions), thereby implementing all or part of the steps in the above-mentioned embodiment method.

[0193] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire a target image, wherein the target image at least includes a face area; Based on the task network corresponding to each predetermined task in the multi-task model, the target image is processed to determine the detection result corresponding to each predetermined task, wherein the multi-task model is at least used for image processing tasks and includes task networks corresponding to multiple predetermined tasks respectively; The multi-task model is trained by the following steps: Obtaining a comprehensive training set corresponding to the multi-task model, the comprehensive training set at least including image samples corresponding to each of the task networks, at least one training label corresponding to the image samples, and a task matching identifier corresponding to the training label, the task matching identifier being used to identify the validity of each of the training labels for each of the predetermined tasks; For each task network, determining a valid label corresponding to the task network and a valid sample corresponding to the valid label in each training label according to the task matching identifier; Determine the target loss corresponding to each of the task networks according to the valid samples and the valid labels; Determining a joint loss according to target losses corresponding to each of the task networks; Based on the joint loss, the multi-task model is trained.

2. The method according to claim 1, characterized in that: The predetermined task includes one or more combinations of the following tasks: a driver face detection task, a driver distraction state detection task, a driver fatigue state detection task, a driver dangerous action detection task, and a driver face occlusion detection task.

3. The method according to claim 1, characterized in that The determining of the joint loss according to the target losses respectively corresponding to the task networks comprises: Based on the target losses and preset weights respectively corresponding to the task networks, weighted summation is performed on the target losses to determine the joint loss, wherein the preset weights are determined based on at least the task type corresponding to the task network.

4. The method according to claim 1 or 3, characterized in that: The training labels include numerical labels; Determining the target loss corresponding to each of the task networks according to the valid samples and the valid labels includes: For each numerical label, normalize and encode the numerical label to determine a normalized numerical value corresponding to the numerical label; According to the valid samples and the normalized values, target losses corresponding to the task networks are determined.

5. The method according to claim 4, characterized in that The image sample includes a detected target area; The step of normalizing and encoding the numerical label to determine the normalized numerical value corresponding to the numerical label includes: According to the size of the detected target area, the predetermined adjustment parameter and the adjustment factor, the numerical label is normalized and encoded to determine the normalized numerical value corresponding to the numerical label.

6. The method according to claim 1, characterized in that The method further comprises: obtaining at least one original sample; According to the sample type corresponding to each of the original samples and the preset enhancement mode corresponding to each of the sample types, data enhancement is performed on each of the original samples to determine an enhanced data set; The comprehensive training set is generated based on each of the original samples and the enhanced data set.

7. An image processing device, characterized in that: The device comprises: A target image acquisition module is configured to acquire a target image, wherein the target image at least includes a face area; A detection result determination module is configured to execute a task network corresponding to each predetermined task in a multi-task model, process the target image, and determine the detection result corresponding to each predetermined task, wherein the multi-task model is at least used for image processing tasks, and the multi-task model includes task networks corresponding to multiple predetermined tasks respectively; The multi-task model is trained through the following modules: A comprehensive training set acquisition module is configured to execute acquisition of a comprehensive training set corresponding to a multi-task model, wherein the multi-task model is at least used for image processing tasks, wherein the multi-task model includes task networks corresponding to a plurality of predetermined tasks, wherein the comprehensive training set includes at least image samples corresponding to each of the task networks, at least one training label corresponding to the image samples, and a task matching identifier corresponding to the training label, wherein the task matching identifier is used to identify the validity of each of the training labels for each of the predetermined tasks; A training data screening module is configured to perform, for each task network, determining, in each training label, a valid label corresponding to the task network and a valid sample corresponding to the valid label according to the task matching identifier; A target loss determination module is configured to determine the target loss corresponding to each of the task networks according to the valid samples and the valid labels; A joint loss determination module is configured to determine the joint loss according to the target losses respectively corresponding to the task networks; The training module is configured to perform training of the multi-task model based on the joint loss.

8. An electronic device, comprising a memory and a processor, characterized in that: The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images