Image processing method and device, electronic equipment and readable storage medium

Abstract

The embodiment of the invention provides an image processing method and device, electronic equipment and a readable storage medium, and relates to the technical field of computers. According to the embodiment of the invention, after the user authorizes to obtain information and starts a detection function, the shared feature map can be determined in a one-time feature extraction mode, and the target extraction network corresponding to each predetermined task extracts and selects own required features from the shared feature map for processing. Therefore, compared with a conventional mode of independently setting a detection model, the method can effectively reduce the number of times of feature extraction, reduces the operation pressure of the electronic equipment on the premise of meeting the accuracy requirement, and achieves the purpose of simply and efficiently executing each image processing task.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image processing method, device, electronic device and readable storage medium. Background Art

[0002] At present, with the development of computer technology, people can perform multiple image processing tasks through one electronic device to achieve the purpose of comprehensive detection.

[0003] In the related art, generally, an image processing model is set up for each image processing task, and the corresponding image processing task is processed separately according to the image processing model. However, since each image processing model needs to occupy a certain amount of computing resources, the related art often has problems such as waste of computing resources and excessive use of computing resources during image processing. Therefore, how to simply and efficiently perform each image processing task is a problem that needs to be solved urgently. Summary of the invention

[0004] In view of this, embodiments of the present application provide an image processing method, apparatus, electronic device, and readable storage medium to simply and efficiently perform various image processing tasks.

[0005] In a first aspect, an image processing method is provided, the method comprising:

[0006] Get the target image.

[0007] Feature extraction is performed on the target image to determine a shared feature map corresponding to the target image, where the shared feature map includes image features required for multiple predetermined tasks.

[0008] Based on the target extraction network corresponding to each of the predetermined tasks, feature extraction is performed on the shared feature map to determine the target feature maps corresponding to each of the predetermined tasks.

[0009] Based on the target feature map corresponding to each of the predetermined tasks, the detection results corresponding to each of the predetermined tasks are determined.

[0010] In some embodiments, extracting features from the target image and determining a shared feature map corresponding to the target image comprises:

[0011] Perform face detection on the target image to determine a face region image of the target image.

[0012] Feature extraction is performed on the face region image to determine a shared feature map corresponding to the target image.

[0013] In some embodiments, extracting features from the face region image to determine a shared feature map corresponding to the target image includes:

[0014] The face region image is expanded and cropped according to the size of the face region image to determine an expanded and cropped region image.

[0015] Feature extraction is performed on the expanded cropped area image to determine a shared feature map corresponding to the target image.

[0016] In some embodiments, the expanded cropped region image includes a limb region image.

[0017] The performing feature extraction on the expanded cropped area image to determine the shared feature map corresponding to the target image comprises:

[0018] At least one local area image corresponding to the expanded cropped area image is determined.

[0019] Multi-channel feature extraction is performed based on the expanded cropped area image and each of the local area images to determine a shared feature map corresponding to the target image.

[0020] In some embodiments, performing multi-channel feature extraction based on the expanded cropped area image and each of the local area images to determine a shared feature map corresponding to the target image includes:

[0021] Determine the face position in the expanded cropped area image.

[0022] A position mask is generated based on the face position.

[0023] Multi-channel feature extraction is performed based on the expanded cropped area image, each of the local area images and the position mask to determine a shared feature map corresponding to the target image.

[0024] In some embodiments, the target extraction network corresponding to each of the predetermined tasks and the main extraction network for extracting the shared feature map are network structures in a multi-task model.

[0025] The method further comprises:

[0026] A comprehensive training set corresponding to the multi-task model is obtained, where the comprehensive training set at least includes a training sample and at least one training label corresponding to the training sample.

[0027] The multi-task model is jointly trained based on the comprehensive training set.

[0028] In response to the convergence of the multi-task model, the model parameters of the main extraction network are kept unchanged, and at least part of the training samples in the comprehensive training set are input into the multi-task model for processing to adjust the model parameters of each of the target extraction networks.

[0029] In some embodiments, the predetermined tasks include at least a face detection task, a driver distraction state detection task, a driver fatigue state detection task, a driver dangerous action detection task, and a driver face occlusion detection task.

[0030] In some embodiments, the method further comprises:

[0031] The accuracy of each of the test results is tested, and the target result is screened out from the test results.

[0032] An image processing result set is generated according to the target result.

[0033] Output the image processing result set.

[0034] In a second aspect, an image processing device is provided, the device comprising:

[0035] The target image acquisition module is configured to execute acquisition of the target image.

[0036] The shared feature determination module is configured to perform feature extraction on the target image and determine a shared feature map corresponding to the target image, wherein the shared feature map includes image features required for multiple predetermined tasks.

[0037] The target feature determination module is configured to perform feature extraction on the shared feature map based on the target extraction network corresponding to each of the predetermined tasks, and determine the target feature map corresponding to each of the predetermined tasks.

[0038] The scheduled task execution module is configured to execute the target feature map corresponding to each of the scheduled tasks and determine the detection results corresponding to each of the scheduled tasks.

[0039] In some embodiments, the shared feature determination module is specifically configured to execute:

[0040] Perform face detection on the target image to determine a face region image of the target image.

[0041] Feature extraction is performed on the face region image to determine a shared feature map corresponding to the target image.

[0042] In some embodiments, the shared feature determination module is specifically configured to execute:

[0043] The face region image is expanded and cropped according to the size of the face region image to determine an expanded and cropped region image.

[0044] Feature extraction is performed on the expanded cropped area image to determine a shared feature map corresponding to the target image.

[0045] In some embodiments, the expanded cropped region image includes a limb region image.

[0046] The shared feature determination module is specifically configured to execute:

[0047] At least one local area image corresponding to the expanded cropped area image is determined.

[0048] Multi-channel feature extraction is performed based on the expanded cropped area image and each of the local area images to determine a shared feature map corresponding to the target image.

[0049] In some embodiments, the shared feature determination module is specifically configured to execute:

[0050] Determine the face position in the expanded cropped area image.

[0051] A position mask is generated based on the face position.

[0052] Multi-channel feature extraction is performed based on the expanded cropped area image, each of the local area images and the position mask to determine a shared feature map corresponding to the target image.

[0053] In some embodiments, the target extraction network corresponding to each of the predetermined tasks and the main extraction network for extracting the shared feature map are network structures in a multi-task model.

[0054] The device also includes:

[0055] The comprehensive training set acquisition module is configured to execute acquisition of a comprehensive training set corresponding to the multi-task model, wherein the comprehensive training set at least includes a training sample and at least one training label corresponding to the training sample.

[0056] The joint training module is configured to perform joint training on the multi-task model based on the comprehensive training set.

[0057] The fine-tuning module is configured to execute, in response to the convergence of the multi-task model, keeping the model parameters of the main extraction network unchanged, and inputting at least part of the training samples in the comprehensive training set into the multi-task model for processing to adjust the model parameters of each of the target extraction networks.

[0058] In some embodiments, the predetermined tasks include at least a face detection task, a driver distraction state detection task, a driver fatigue state detection task, a driver dangerous action detection task, and a driver face occlusion detection task.

[0059] In some embodiments, the apparatus further comprises:

[0060] The accuracy detection module is configured to perform accuracy detection on each of the detection results and filter out the target result from each of the detection results.

[0061] The image processing result set generating module is configured to generate an image processing result set according to the target result.

[0062] The output module is configured to output the image processing result set.

[0063] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method described in the first aspect.

[0064] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the method described in the first aspect is implemented.

[0065] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the method described in the first aspect.

[0066] The embodiment of the present application can determine the shared feature map through a one-time feature extraction method, and the target extraction network corresponding to each predetermined task extracts and selects the features it needs for processing. Therefore, compared with the conventional method of setting up a detection model separately, the embodiment of the present application can effectively reduce the number of feature extractions (i.e., from multiple feature extractions to one feature extraction). On the premise of meeting the accuracy requirements, it reduces the computing pressure of electronic equipment (especially devices with less computing resources such as camera equipment dedicated to online car-hailing), and achieves the purpose of simply and efficiently executing various image processing tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] The above and other purposes, features and advantages of the embodiments of the present application will become more apparent through the following description of the embodiments of the present application with reference to the accompanying drawings, in which:

[0068] Figure 1 A schematic diagram of the flow of an image processing method according to an embodiment of the present application;

[0069] Figure 2 A flowchart of an image processing method according to an embodiment of the present application;

[0070] Figure 3 A flowchart of another image processing method according to an embodiment of the present application;

[0071] Figure 4 A schematic diagram of a flow chart of another image processing method according to an embodiment of the present application;

[0072] Figure 5 A schematic diagram of a flow chart of another image processing method according to an embodiment of the present application;

[0073] Figure 6 A schematic diagram of a flow chart of another image processing method according to an embodiment of the present application;

[0074] Figure 7 A schematic diagram of the structure of a multi-task model according to an embodiment of the present application;

[0075] Figure 8 A flowchart of another image processing method according to an embodiment of the present application;

[0076] Fig. 9 A flowchart of another image processing method according to an embodiment of the present application;

[0077] Fig.10 A schematic diagram of the structure of another multi-task model according to an embodiment of the present application;

[0078] Fig.11 A schematic diagram of the structure of an image processing device according to an embodiment of the present application;

[0079] Fig.12 A schematic diagram of the structure of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0080] The present application is described below based on embodiments, but the present application is not limited to these embodiments. In the detailed description of the present application below, some specific details are described in detail. It is possible for those skilled in the art to fully understand the present application without the description of these details. In order to avoid confusing the essence of the present application, known methods, processes, flows, components and circuits are not described in detail.

[0081] In addition, persons of ordinary skill in the art will appreciate that the drawings provided herein are for illustration purposes and are not necessarily drawn to scale.

[0082] Unless the context clearly requires otherwise, the words "include", "comprising" and the like throughout this application should be interpreted as including rather than exclusive or exhaustive; that is, as meaning "including but not limited to".

[0083] In the description of this application, it should be understood that the terms "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of this application, unless otherwise specified, the meaning of "plurality" is two or more.

[0084] The solutions described in this specification and in the examples, if they involve the processing of personal information, will be processed on the premise of having a legal basis (such as obtaining the consent of the subject of personal information, or being necessary for the performance of a contract, etc.), and will only be processed within the scope of regulations or agreements. If a user refuses to process personal information other than the necessary information for basic functions, it will not affect the user's use of basic functions.

[0085] At present, with the development of computer technology, people can use one electronic device to perform multiple image processing tasks to achieve the purpose of comprehensive detection. In the relevant technology, generally, an image processing model is set separately for each image processing task, and the corresponding image processing task is processed separately according to the image processing model.

[0086] Taking online car-hailing services as an example, in order to ensure the driving safety of drivers (i.e. online car-hailing drivers), online car-hailing service platforms can collect driver images through online car-hailing dedicated camera equipment after authorization, and the online car-hailing dedicated camera equipment will perform image processing such as driver distraction detection, driver fatigue detection, driver dangerous action detection, and driver face occlusion detection based on the driver image to determine whether the driver is in a safe driving state. Among them, the relevant technology will set up a separate image processing model for each detection task, and each image processing model needs to perform feature extraction and feature processing operations according to task requirements.

[0087] For example, when the related technology processes the driver image through the image processing model of driver distraction state detection, it will extract and process the features such as eyes and head rotation. When the driver image is processed through the image processing model of driver fatigue state detection, it will extract and process the features such as eyes and mouth. When the driver image is processed through the image processing model of driver dangerous action detection, it will extract and process the features such as mouth and ear area.

[0088] However, since each image processing model requires a certain amount of computing resources, the related technologies often have problems such as waste of computing resources and excessive use of computing resources during image processing. For example, in the scenario of the above-mentioned online car-hailing service, the image processing task generally needs to be independently performed by the camera equipment dedicated to the above-mentioned online car-hailing service. However, due to the influence of the hardware resources of the camera equipment itself, the computing resources of the camera equipment are often limited, and it is impossible to perform too many image processing tasks at the same time. In other words, when there are too many image processing tasks to be processed (that is, when there are too many image processing models to be run), the camera equipment will be under greater computing pressure.

[0089] Moreover, the method of performing image detection only through local features is prone to misjudgment and missed judgment, thereby reducing the accuracy of image processing. Therefore, how to perform various image processing tasks simply and efficiently is a problem that needs to be solved urgently.

[0090] In order to solve the above problems, the embodiment of the present application provides an image processing method, which satisfies the image processing requirements of multiple predetermined tasks through a shared feature map determined by a one-time feature extraction, and realizes the simple and efficient execution of each image processing task. Among them, the image processing method of the embodiment of the present application can be applied to electronic devices, and the electronic device can be a terminal or a server. The terminal can be a smart phone, a tablet computer, a vehicle control terminal, a camera device (such as a driving recorder or a camera device dedicated to online car-hailing) or a personal computer (Personal Computer, PC), etc. The server can be a single server, a server cluster configured in a distributed manner, or a cloud server.

[0091] Specifically, Figure 1 As shown, after acquiring the target image 11, the embodiment of the present application can perform feature extraction on the target image 11 based on the main extraction network 12 to determine the shared feature map 13 corresponding to the target image 11, wherein. The shared feature map 13 may include image features required for multiple predetermined tasks, and the predetermined tasks may be various subtasks under a general task. For example, the general task may be the above-mentioned driving safety detection task, and the predetermined tasks may be subtasks such as the face detection task, the driver distraction state detection task, the driver fatigue state detection task, the driver dangerous action detection task, and the driver face occlusion detection task under the driving safety detection task.

[0092] Furthermore, the embodiments of the present application can perform feature extraction on the shared feature graph 13 based on the target extraction networks corresponding to each predetermined task (target extraction network 141-target extraction network 14n, n is a natural number greater than or equal to 2), and determine the target feature graphs (target feature graphs 151-target feature graphs 15n) corresponding to each predetermined task. For example, the target extraction network corresponding to the face detection task can determine the overall features of the face from the shared feature graph 13 to determine the target feature graph corresponding to the face detection task. For another example, the target extraction network corresponding to the driver distraction state detection task can determine the features of the eyes, mouth, ears, or the entire face from the shared feature graph 13 to determine the target feature graph corresponding to the driver distraction state detection task.

[0093] Furthermore, the embodiment of the present application can determine the detection results (detection results 161-detection results 16n) corresponding to each scheduled task based on the target feature graph corresponding to each scheduled task. Among them, the embodiment of the present application can set a corresponding task processing network for each scheduled task, and the task processing network can be a classification network constructed based on a fully connected network, or it can be a regression network constructed based on a fully connected network. After the task processing network corresponding to each scheduled task receives the target feature graph, the target feature graph can be classified or regressed to determine the detection result (classification result or regression result).

[0094] In this process, since the embodiment of the present application can determine the shared feature map through a one-time feature extraction method, and the target extraction network corresponding to each predetermined task extracts and selects the features needed by itself for processing, compared with the conventional method of setting up a detection model separately, the embodiment of the present application can effectively reduce the number of feature extractions (that is, from n feature extractions to 1 feature extraction). On the premise of meeting the accuracy requirements, it reduces the computing pressure of electronic equipment (especially devices with less computing resources such as camera equipment dedicated to online ride-hailing), and achieves the purpose of simply and efficiently executing various image processing tasks.

[0095] The image processing method provided by the embodiment of the present application will be described in detail below in conjunction with specific implementation methods. Figure 2 As shown, the image processing method of the embodiment of the present application may specifically include the following steps:

[0096] In step S110, a target image is acquired.

[0097] Among them, the target image can be directly captured by a camera unit set up in the electronic device itself (such as the camera of a smartphone, the camera of a driving recorder, or the camera of a camera device dedicated to an online car-hailing service), or it can be captured by a camera device external to the electronic device (wired connection or wireless connection).

[0098] In step S120, feature extraction is performed on the target image to determine a shared feature map corresponding to the target image.

[0099] Among them, the shared feature map may include image features required for multiple predetermined tasks, and the predetermined tasks may be various subtasks under a general task. For example, the general task may be the above-mentioned driving safety detection task, and the predetermined tasks may be subtasks such as face detection task, driver distraction state detection task, driver fatigue state detection task, driver dangerous action detection task, and driver face occlusion detection task under the driving safety detection task. For another example, the general task may also be an illegally parked vehicle detection task, and the predetermined tasks may be subtasks such as vehicle detection task, parking space detection task, and vehicle state detection task under the illegally parked vehicle detection task.

[0100] In addition, the embodiment of the present application can perform a one-time feature extraction on the target image through a feature extraction model to determine the shared feature map corresponding to the target image. The feature extraction model can be a pre-set lightweight network (such as ShuffleNetV2 0.5x) or other applicable feature extraction models. Optionally, the embodiment of the present application can also add an attention module to the feature extraction model used to determine the shared feature map to improve the effect of feature extraction. For example, the above-mentioned attention module can be a garbage collection (GC) module, a compression and excitation network (Squeeze-and-Excitation, SE) module, a convolutional attention mechanism (Convolutional Block Attention Module, CBAM) module or other applicable attention modules.

[0101] In an optional implementation, the predetermined tasks of the embodiment of the present application may include at least a face detection task, a driver distraction state detection task, a driver fatigue state detection task, a driver dangerous action detection task, and a driver face occlusion detection task.

[0102] That is to say, through comprehensive detection of face detection tasks, driver distraction status detection tasks, driver fatigue status detection tasks, driver dangerous action detection tasks and driver face occlusion detection tasks, the embodiments of the present application can effectively determine whether the driver is in a safe driving state, thereby improving the driver's driving safety.

[0103] In an optional implementation, the shared feature map may be a feature map of a face region, specifically, Figure 3 As shown, the above step S120 may include the following steps:

[0104] In step S121, face detection is performed on the target image to determine a face region image of the target image.

[0105] Among them, the embodiment of the present application can detect the face in the target image through the face detection model, and identify the face in the target image through the face detection frame. That is to say, the area identified by the face detection frame is the face area image of the embodiment of the present application.

[0106] In step S122, feature extraction is performed on the face region image to determine a shared feature map corresponding to the target image.

[0107] That is to say, the embodiment of the present application can make the shared feature map only include the features of the face area image by first performing face detection and then feature extraction. Similarly, if the embodiment of the present application performs image processing for other types of targets (such as four-wheeled vehicles, two-wheeled vehicles, etc.), the embodiment of the present application can first perform target detection on the target to be detected to determine the area image of the target to be detected, and then perform feature extraction on the area image of the target to be detected to determine the shared feature map corresponding to the target image. In this way, the embodiment of the present application can exclude the influence of factors such as background in the process of feature extraction, so that the image processing process of the embodiment of the present application is more targeted.

[0108] In an optional implementation, after determining the face region image, the embodiment of the present application may further expand and crop the face region image. Specifically, the above step S122 may include the following steps:

[0109] In step A01, the face region image is expanded and cropped according to the size of the face region image to determine an expanded and cropped region image.

[0110] In one case, when extracting features from an image, it is often necessary to adjust (resize) the size of the input image (i.e., the face area image) so that the face area image is resized to a square size. At this time, if the input face area image is not of a square size, the face area image will be deformed to a certain extent after being resized, thereby causing the original information in the face area image to be changed (for example, the human eye will be more inclined to be closed after being elongated), thereby affecting the effect of image processing. In response to this situation, an embodiment of the present application can expand and crop the face area image in a direction perpendicular to the long side of the face area image so that the expanded and cropped face area image is a square area, thereby ensuring that the original information in the face area image is not changed.

[0111] In another case, in the process of performing tasks such as driving safety detection, in addition to obtaining information included in the face area image, the embodiment of the present application can also input the limb area as supplementary information into the feature extraction model for feature extraction, thereby increasing the dimension of information acquisition and improving the accuracy of image processing tasks.

[0112] In step A02, feature extraction is performed on the expanded cropped area image to determine a shared feature map corresponding to the target image.

[0113] For example, Figure 4 As shown, after determining the target image 41, the embodiment of the present application can perform face detection on the target image 41 to determine the face area image 42 of the target image 41. Figure 4 It can be seen that the face region image 42 is a rectangular detection frame. If feature extraction is performed directly on the face region image 42, the face region image 42 will be deformed to a certain extent after being resized, thereby causing the original information in the face region image 42 to be changed.

[0114] In view of this situation, the embodiment of the present application can expand and crop the face area image 42 to determine the expanded cropped area image 43 or the expanded cropped area image 44. Among them, the expanded cropped area image 43 is an image obtained by expanding and cropping only the face area, which can ensure that the original information in the face area image is not changed after resizing. The expanded cropped area image 44 is an image that includes both the face area and the limb area, which can increase the dimension of information acquisition and improve the accuracy of image processing tasks (such as the above-mentioned driving safety detection tasks).

[0115] In an optional embodiment, the expanded cropped region image includes a limb region image (eg Figure 4 In the case of the expanded cropping area image 44), the above step A02 may include the following steps:

[0116] In step A021, at least one local area image corresponding to the expanded cropped area image is determined.

[0117] The local area image is a part of the expanded cropped area image. The embodiment of the present application can randomly zoom in on the expanded cropped area image to determine the local area image. The embodiment of the present application can also first perform image recognition on the expanded cropped area image to determine multiple recognition targets in the expanded cropped area image (such as recognition targets such as hand area, arm area, and face area), and use each recognition target as a reference to determine the local enlarged image corresponding to each recognition target, each local enlarged image being a local area image.

[0118] In step A022, multi-channel feature extraction is performed based on the expanded cropped area image and each local area image to determine a shared feature map corresponding to the target image.

[0119] Among them, the number of channels for multi-channel feature extraction is the same as the total number of the expanded cropped area image and each local area image. For example, if the expanded cropped area image is 1 and the number of local area images is 5, the total number of expanded cropped area images and each local area image is 6. Furthermore, after determining each local area image, the embodiment of the present application can resize the expanded cropped area image and each local area image to the same size (for example, 160 pixels * 160 pixels), and input each resized image into the feature extraction model, so that the feature extraction model performs multi-channel feature extraction, thereby determining the shared feature map corresponding to the target image.

[0120] Taking the above-mentioned expanded cropping area image 44 as an example, Figure 5 As shown, after determining the extended cropped region image 44, the embodiment of the present application can determine each local region image corresponding to the extended cropped region image, and resize the extended cropped region image and each local region image to the same size to obtain an image set 51. The image set 51 includes one extended cropped region image and three local region images.

[0121] Furthermore, the embodiment of the present application may input each image in the image set 51 into the feature extraction model 52, so that the feature extraction model 52 performs multi-channel feature extraction, thereby determining the shared feature map 53 corresponding to the target image.

[0122] The embodiment of the present application can increase the dimension of information acquisition and improve the accuracy of image processing tasks (such as the above-mentioned driving safety detection tasks) by determining local area images and multi-channel feature extraction.

[0123] In an optional implementation, in order to avoid the influence of the local area image on the shared feature map, the embodiment of the present application may also add a position mask during multi-channel feature extraction. Specifically, the above step A022 may include the following steps:

[0124] Step A0221, determine the face position in the expanded cropped area image.

[0125] Among them, the face position of the embodiment of the present application can be determined based on the above-mentioned face detection process (for example, step S121).

[0126] Step A0222, generating a position mask based on the face position.

[0127] The position mask is used to characterize the face position in the expanded cropped area image. In practical applications, the local area image may have a certain impact on the shared feature map. For example, if the driver's clothes contain face patterns, human action patterns, or other patterns that may affect the shared feature map, the feature extraction model may mistake the pattern for the driver and extract the pattern as the driver's feature, thereby affecting the features in the final shared feature map.

[0128] Therefore, in order to avoid the influence of the local area image on the shared feature map, the embodiment of the present application can generate a position mask corresponding to the face position to identify the face position in the expanded cropped area image. In this way, even if the driver's clothes contain face patterns, character action patterns or other patterns that may affect the shared feature map, the feature extraction model can exclude the above patterns according to the position mask, thereby avoiding unnecessary influence and improving the accuracy of image processing.

[0129] Step A0223, performing multi-channel feature extraction based on the expanded cropped area image, each local area image and the position mask to determine the shared feature map corresponding to the target image.

[0130] For example, Figure 6 As shown, after acquiring the target image 61, the embodiment of the present application can determine the extended cropped area image 62 through face detection and extended cropping and other processing, and further determine multiple local area images corresponding to the extended cropped area image 62 to obtain an image set 63. In addition, after determining the face position, the embodiment of the present application can also generate a position mask 64 for representing the face position according to the face position, and the position mask 64 can be used to identify the face position in the extended cropped area image.

[0131] Furthermore, after determining the image set 63 (including the expanded cropped area image 62 and its corresponding multiple local area images) and the position mask 64, the embodiment of the present application can input the image set 63 and the position mask 64 together into the feature extraction model 65, so that the feature extraction model 65 performs multi-channel feature extraction on the image set 63 and the position mask 64, and determines the shared feature map 66 corresponding to the target image 61. The number of channels for feature extraction of the feature extraction model 65 is the sum of the number of the expanded cropped area image 62, the multiple local area images corresponding to the expanded cropped area image 62, and the position mask 64. Figure 6 For example, the number of channels for feature extraction of feature extraction model 65 is 1+3+1=5.

[0132] By adding a position mask to the multi-channel feature extraction process, the embodiment of the present application can identify the face position in the expanded cropped area image. In this way, even if the driver's clothes contain face patterns, human action patterns, or other patterns that may affect the shared feature map, the feature extraction model can exclude the above patterns based on the position mask, thereby avoiding unnecessary influence and improving the accuracy of image processing.

[0133] In step S130, feature extraction is performed on the shared feature graph based on the target extraction network corresponding to each predetermined task, and the target feature graph corresponding to each predetermined task is determined.

[0134] Among them, the target extraction network of the embodiment of the present application can be a lightweight classification / regression head (LiteHead), and LiteHead can be an extraction network built based on a deep convolution (DW) network, which can extract the features of the current scheduled task (i.e., the scheduled task corresponding to LiteHead) from the shared feature map, thereby determining the target feature map corresponding to the current scheduled task. In other words, the embodiment of the present application can set a LiteHead for each scheduled task to determine the target feature map corresponding to each scheduled task.

[0135] In step S140, based on the target feature graph corresponding to each predetermined task, the detection result corresponding to each predetermined task is determined.

[0136] Among them, the detection results can be the classification results or regression results corresponding to each predetermined task. Taking the overall task of driving safety detection as an example, the detection results can be the driver's face position, whether the driver is distracted, whether the driver is tired, whether the driver has dangerous actions (such as answering or making phone calls or smoking), and whether the driver's face is blocked.

[0137] The embodiment of the present application can determine the shared feature map through a one-time feature extraction method, and the target extraction network corresponding to each predetermined task extracts and selects the features it needs for processing. Therefore, compared with the conventional method of setting up a detection model separately, the embodiment of the present application can effectively reduce the number of feature extractions (i.e., from multiple feature extractions to one feature extraction). On the premise of meeting the accuracy requirements, it reduces the computing pressure of electronic equipment (especially devices with less computing resources such as camera equipment dedicated to online car-hailing), and achieves the purpose of simply and efficiently executing various image processing tasks.

[0138] In an optional implementation, the target extraction network corresponding to the predetermined task and the main extraction network for extracting the shared feature map are network structures in a multi-task model.

[0139] That is to say, the embodiments of the present application can synchronously process multiple predetermined tasks through one model, reduce the number of models set in the electronic device, and improve the efficiency of image processing.

[0140] For example, Figure 7 As shown, the multi-task model of an embodiment of the present application may include a target detection network 71, an expanded cropping network 72, a main extraction network 73, multiple target extraction networks (target extraction network 741-target extraction network 74m, m is a natural number greater than or equal to 2) and a task processing network (task processing network 751-task processing network 75m) corresponding to each target extraction network.

[0141] Among them, the target detection network 71 can be used to perform target detection tasks (such as face detection, vehicle detection and other tasks). The expansion and cropping network 72 can be used to expand the cropping process and determine the expanded cropping area image. The main extraction network 73 can be used to determine the shared feature map. The target extraction network 741-target extraction network 74m can be used to determine the target feature map corresponding to each predetermined task. The task processing network 751-task processing network 75m can be used to process the corresponding target feature map to determine the detection result. Among them, the task processing network can be a classification network built based on a fully connected network (for example, a network for detecting whether the driver is distracted or a network for detecting whether the driver is tired), or it can be a regression network built based on a fully connected network (for example, a network for detecting eyelid distance).

[0142] Taking the online car-hailing service as an example, in order to ensure the driving safety of the driver (i.e., the online car-hailing driver), the online car-hailing service platform's dedicated camera equipment can collect the driver's image and input the driver's image as the target image into the multi-task model, so that after the multi-task model obtains the target image, it performs face detection on the target image through the target detection network 71 to determine the face area image of the target image. Furthermore, the dedicated camera equipment for online car-hailing can perform expansion and cropping processing on the face area image based on the expansion and cropping network 72 to determine the expansion and cropping area image. Among them, the expansion and cropping area image can include only the face area image, or it can include the face area image and the limb area image at the same time.

[0143] Furthermore, taking the case where the expanded cropped area image includes both the face area image and the limb area image, the online car-hailing dedicated camera device can determine at least one local area image corresponding to the expanded cropped area image, and a position mask corresponding to the face position in the expanded cropped area image. Furthermore, the online car-hailing dedicated camera device can input the expanded cropped area image, each local area image, and the position mask corresponding to the face position into the main extraction network 73, so that the main extraction network 73 performs multi-channel feature extraction and determines a shared feature map.

[0144] Furthermore, the online car-hailing dedicated camera equipment can input the shared feature map into each target extraction network (target extraction network 741-target extraction network 74m), so that each target extraction network selects to extract the features of its own concern (i.e., target feature map), and inputs the target feature map into the corresponding task processing network, so that the task processing network processes the target feature map and outputs the detection result. In this process, since the embodiment of the present application can perform a one-time feature extraction through the main extraction network 73 to determine the shared feature map, and the target extraction network corresponding to each predetermined task extracts and selects the features it needs for processing, compared with the conventional method of setting up a detection model separately, the embodiment of the present application can effectively reduce the number of feature extractions (i.e., from m feature extractions to 1 feature extraction), and on the premise of meeting the accuracy requirements, reduce the computing pressure of the online car-hailing dedicated camera equipment, and achieve the purpose of simply and efficiently executing each image processing task.

[0145] Furthermore, for the training process of the above multi-task model, Figure 8 As shown, the embodiment of the present application can also perform the following steps:

[0146] In step S210, a comprehensive training set corresponding to the multi-task model is obtained.

[0147] The comprehensive training set at least includes training samples and at least one training label corresponding to the training samples.

[0148] In step S220, the multi-task model is jointly trained based on the comprehensive training set.

[0149] Specifically, the embodiment of the present application can input the training samples in the comprehensive training set into the multi-task model, determine the output results of each target extraction network, and determine the target loss of the target extraction network based on the output results and the training labels. Further, after determining the target loss of each target extraction network, the embodiment of the present application can perform a weighted summation of the target losses to determine the joint loss, and adjust the model parameters of the multi-task model as a whole based on the joint loss, thereby achieving the purpose of joint training.

[0150] In one case, each training sample in the comprehensive training set has training labels corresponding to all predetermined tasks, that is, each training sample can be directly used to train all target extraction networks. In this case, the embodiment of the present application can directly input the training samples into the multi-task model for joint training.

[0151] In another case, each training sample in the comprehensive training set does not have training labels corresponding to all scheduled tasks. For example, the comprehensive training set can be a training set directly combined from the training sets corresponding to each scheduled task. In this case, each scheduled task has invalid training labels (i.e., unsuitable training samples) in the comprehensive training set. At this point, the embodiment of the present application can set an invalid mark for the above-mentioned invalid training labels for each scheduled task. So that in the process of joint training, the target extraction network corresponding to each scheduled task can automatically screen the training labels according to the invalid mark, without the need for additional annotation of the training samples, saving the time of creating a comprehensive training set and improving the training efficiency.

[0152] In step S230, in response to the convergence of the multi-task model, the model parameters of the main extraction network are kept unchanged, and at least part of the training samples in the comprehensive training set are input into the multi-task model for processing to adjust the model parameters of each target extraction network.

[0153] Specifically, the embodiment of the present application can input all or part of the training samples in the comprehensive training set into the multi-task model, and determine the target loss corresponding to each target extraction network based on the results output by each target extraction network. Furthermore, the embodiment of the present application can keep the model parameters of the main extraction network unchanged, and adjust the model parameters of each target extraction network respectively through the target loss corresponding to each target extraction network. In other words, after the overall joint training, the embodiment of the present application can fine-tune each target extraction network individually to further improve the feature extraction capability of the multi-task model.

[0154] In addition, when fine-tuning each target extraction network, the embodiment of the present application can also sample new training samples (that is, the training samples fine-tuned in the embodiment of the present application are not limited to the training samples in the comprehensive training set) to increase the number of samples and improve the training effect.

[0155] In an optional implementation, the present application embodiment can also screen the test results, specifically, Fig. 9 As shown, the embodiment of the present application may also include the following steps:

[0156] In step S310, accuracy test is performed on each test result, and target results are screened out from each test result.

[0157] Among them, the embodiments of the present application can screen each detection result based on a pre-trained machine learning model (such as a machine learning model set in the cloud) to filter out abnormal detection results.

[0158] In step S320, an image processing result set is generated according to the target result.

[0159] Among them, the target results in the image processing result set are the detection results retained after screening and filtering.

[0160] In step S330, an image processing result set is output.

[0161] Among them, the method of outputting the image processing result set in the embodiment of the present application may include one or more combinations of text prompts, voice broadcasts, and video presentations.

[0162] For example, Fig.10 As shown, the embodiment of the present application can set a machine learning model 1001 after the task processing network (task processing network 751-task processing network 75m) of the multi-task model. After each task processing network determines the test result, the embodiment of the present application can input each test result into the machine learning model 1001, so that the machine learning model 1001 screens each test result to filter out abnormal test results and retain the target result. Among them, the machine learning model 1001 can be a classification model for the test result, and its classification results can include normal test results and abnormal test results. In the process of training, the embodiment of the present application can train the machine learning model 1001 based on historical data and the training labels corresponding to the historical data, so that the machine learning model 1001 can accurately identify the abnormal data in each test result.

[0163] Furthermore, after determining the target result, the embodiment of the present application can generate an image processing result set based on the target result, and output the image processing result set in the form of text prompts, voice broadcasts, or video prompts to remind the user of the results in the image processing result set.

[0164] It should be noted that the machine learning model 1001 can also be a machine learning model set up in the cloud. In this case, after determining each detection result, the embodiment of the present application can report each detection result to the machine learning model (i.e., cloud-based machine learning model) in the cloud server (e.g., online car-hailing service platform) through the network, so that the cloud-based machine learning model can filter out the target result from each detection result. Furthermore, after determining the target result, the image processing result set can be directly output by the cloud device (e.g., cloud server), or the local device can receive the target result and output the image processing result set.

[0165] In this process, compared with the conventional method of setting up a detection model separately, the embodiment of the present application can effectively reduce the number of feature extractions (i.e., from multiple feature extractions to one feature extraction). On the premise of meeting the accuracy requirements, it reduces the computing pressure of electronic devices (especially devices with less computing resources such as camera equipment dedicated to online ride-hailing), thereby achieving the purpose of simply and efficiently executing various image processing tasks.

[0166] Based on the same technical concept, the embodiment of the present application also provides an image processing device, such as Fig.11 As shown, the device includes: a target image acquisition module 1101, a shared feature determination module 1102, a target feature determination module 1103 and a predetermined task execution module 1104.

[0167] The target image acquisition module 1101 is configured to acquire a target image.

[0168] The shared feature determination module 1102 is configured to perform feature extraction on the target image and determine a shared feature map corresponding to the target image, wherein the shared feature map includes image features required for multiple predetermined tasks.

[0169] The target feature determination module 1103 is configured to perform feature extraction on the shared feature graph based on the target extraction network corresponding to each of the predetermined tasks, and determine the target feature graph corresponding to each of the predetermined tasks.

[0170] The scheduled task execution module 1104 is configured to execute the target feature map corresponding to each of the scheduled tasks and determine the detection results corresponding to each of the scheduled tasks.

[0171] In some embodiments, the shared feature determination module 1102 is specifically configured to perform:

[0172] Perform face detection on the target image to determine a face region image of the target image.

[0173] Feature extraction is performed on the face region image to determine a shared feature map corresponding to the target image.

[0174] In some embodiments, the shared feature determination module 1102 is specifically configured to perform:

[0175] The face region image is expanded and cropped according to the size of the face region image to determine an expanded and cropped region image.

[0176] Feature extraction is performed on the expanded cropped area image to determine a shared feature map corresponding to the target image.

[0177] In some embodiments, the expanded cropped region image includes a limb region image.

[0178] The shared feature determination module 1102 is specifically configured to execute:

[0179] At least one local area image corresponding to the expanded cropped area image is determined.

[0180] Multi-channel feature extraction is performed based on the expanded cropped area image and each of the local area images to determine a shared feature map corresponding to the target image.

[0181] In some embodiments, the shared feature determination module 1102 is specifically configured to perform:

[0182] Determine the face position in the expanded cropped area image.

[0183] A position mask is generated based on the face position.

[0184] Multi-channel feature extraction is performed based on the expanded cropped area image, each of the local area images and the position mask to determine a shared feature map corresponding to the target image.

[0185] In some embodiments, the target extraction network corresponding to each of the predetermined tasks and the main extraction network for extracting the shared feature map are network structures in a multi-task model.

[0186] The device also includes:

[0187] The comprehensive training set acquisition module is configured to execute acquisition of a comprehensive training set corresponding to the multi-task model, wherein the comprehensive training set at least includes a training sample and at least one training label corresponding to the training sample.

[0188] The joint training module is configured to perform joint training on the multi-task model based on the comprehensive training set.

[0189] The fine-tuning module is configured to execute, in response to the convergence of the multi-task model, keeping the model parameters of the main extraction network unchanged, and inputting at least part of the training samples in the comprehensive training set into the multi-task model for processing to adjust the model parameters of each of the target extraction networks.

[0190] In some embodiments, the predetermined tasks include at least a face detection task, a driver distraction state detection task, a driver fatigue state detection task, a driver dangerous action detection task, and a driver face occlusion detection task.

[0191] In some embodiments, the apparatus further comprises:

[0192] The accuracy detection module is configured to perform accuracy detection on each of the detection results and filter out the target result from each of the detection results.

[0193] The image processing result set generating module is configured to generate an image processing result set according to the target result.

[0194] The output module is configured to output the image processing result set.

[0195] The embodiment of the present application can determine the shared feature map through a one-time feature extraction method, and the target extraction network corresponding to each predetermined task extracts and selects the features it needs for processing. Therefore, compared with the conventional method of setting up a detection model separately, the embodiment of the present application can effectively reduce the number of feature extractions (i.e., from multiple feature extractions to one feature extraction). On the premise of meeting the accuracy requirements, it reduces the computing pressure of electronic equipment (especially devices with less computing resources such as camera equipment dedicated to online car-hailing), and achieves the purpose of simply and efficiently executing various image processing tasks.

[0196] Fig.12 Schematic diagram of an electronic device according to an embodiment of the present application. Fig.12 As shown, Fig.12 The electronic device shown is a general address query device, which includes a general computer hardware structure, which includes at least a processor 1201 and a memory 1202. The processor 1201 and the memory 1202 are connected via a bus 1203. The memory 1202 is suitable for storing instructions or programs executable by the processor 1201. The processor 1201 can be an independent microprocessor or a collection of one or more microprocessors. Thus, the processor 1201 executes the instructions stored in the memory 1202, thereby executing the method flow of the embodiment of the present application as described above to realize the processing of data and the control of other devices. The bus 1203 connects the above-mentioned multiple components together, and at the same time connects the above-mentioned components to the display controller 1204 and the display device and the input / output (I / O) device 1205. The input / output (I / O) device 1205 can be a mouse, a keyboard, a modem, a network interface, a touch input device, a somatosensory input device, a printer, and other devices known in the art. Typically, the input / output device 1205 is connected to the system through an input / output (I / O) controller 1206.

[0197] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices (equipment) or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the present application may adopt a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0198] The present application is described with reference to flowcharts of methods, apparatuses (devices) and computer program products according to embodiments of the present application. It should be understood that each process in the flowchart can be implemented by computer program instructions.

[0199] These computer program instructions may be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device that implements the process Figure 1 A function specified in a process or multiple processes.

[0200] These computer program instructions may also be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce the instructions for implementing the process Figure 1 A device that specifies functions in a process or multiple processes.

[0201] Another embodiment of the present application relates to a non-volatile storage medium for storing a computer-readable program, wherein the computer-readable program is used for a computer to execute part or all of the above method embodiments.

[0202] That is, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiment method can be completed by specifying relevant hardware through a program, and the program is stored in a storage medium, including several instructions for a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0203] Another embodiment of the present application relates to a computer program product, including a computer program / instruction, which can implement part or all of the above method embodiments when the computer program / instruction is executed by a processor.

[0204] That is, those skilled in the art can understand that the embodiments of the present application can specify relevant hardware (including the processor itself) through the processor executing a computer program product (computer program / instructions), thereby implementing all or part of the steps in the above-mentioned embodiment method.

[0205] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Get the target image; Extracting features from the target image to determine a shared feature map corresponding to the target image, wherein the shared feature map includes image features required for a plurality of predetermined tasks; Based on the target extraction network corresponding to each of the predetermined tasks, feature extraction is performed on the shared feature map to determine the target feature map corresponding to each of the predetermined tasks; Based on the target feature map corresponding to each of the predetermined tasks, the detection results corresponding to each of the predetermined tasks are determined.

2. The method according to claim 1, characterized in that The extracting features of the target image and determining a shared feature map corresponding to the target image comprises: Performing face detection on the target image to determine a face region image of the target image; Feature extraction is performed on the face region image to determine a shared feature map corresponding to the target image.

3. The method according to claim 2, characterized in that The extracting features of the face region image to determine the shared feature map corresponding to the target image comprises: Expanding and cropping the face region image according to the size of the face region image to determine an expanded and cropped region image; Feature extraction is performed on the expanded cropped area image to determine a shared feature map corresponding to the target image.

4. The method according to claim 3, characterized in that The expanded cropped region image includes a limb region image; The performing feature extraction on the expanded cropped area image to determine the shared feature map corresponding to the target image comprises: Determine at least one local area image corresponding to the expanded cropped area image; Multi-channel feature extraction is performed based on the expanded cropped area image and each of the local area images to determine a shared feature map corresponding to the target image.

5. The method according to claim 4, characterized in that The performing multi-channel feature extraction based on the expanded cropped area image and each of the local area images to determine the shared feature map corresponding to the target image comprises: Determining the position of the face in the expanded cropped area image; generating a position mask based on the face position; Multi-channel feature extraction is performed based on the expanded cropped area image, each of the local area images and the position mask to determine a shared feature map corresponding to the target image.

6. The method according to claim 1, characterized in that The target extraction network corresponding to each of the predetermined tasks and the main extraction network for extracting the shared feature map are the network structures in the multi-task model; The method further comprises: Obtaining a comprehensive training set corresponding to the multi-task model, wherein the comprehensive training set includes at least a training sample and at least one training label corresponding to the training sample; Performing joint training on the multi-task model based on the comprehensive training set; In response to the convergence of the multi-task model, the model parameters of the main extraction network are kept unchanged, and at least part of the training samples in the comprehensive training set are input into the multi-task model for processing to adjust the model parameters of each of the target extraction networks.

7. The method according to claim 1, characterized in that The predetermined tasks include at least a face detection task, a driver distraction state detection task, a driver fatigue state detection task, a driver dangerous action detection task and a driver face occlusion detection task.

8. The method according to claim 1, characterized in that The method further comprises: Performing accuracy testing on each of the test results, and screening out target results from each of the test results; Generate an image processing result set according to the target result; Output the image processing result set.

9. An image processing device, characterized in that: The device comprises: A target image acquisition module is configured to acquire a target image; A shared feature determination module is configured to perform feature extraction on the target image and determine a shared feature map corresponding to the target image, wherein the shared feature map includes image features required for a plurality of predetermined tasks; A target feature determination module is configured to perform feature extraction on the shared feature graph based on the target extraction network corresponding to each of the predetermined tasks, and determine the target feature graph corresponding to each of the predetermined tasks; The scheduled task execution module is configured to execute the target feature map corresponding to each of the scheduled tasks and determine the detection results corresponding to each of the scheduled tasks.

10. An electronic device comprising a memory and a processor, characterized in that: The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

12. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images