Image processing method, processing device, electronic device and storage medium

Abstract

The present application discloses an image processing method, an image processing device, an electronic device and a computer-readable storage medium, wherein the image processing method comprises: extracting features of an image to be processed based on a backbone network of a pre-trained image processing model to obtain image features; inputting the image features into at least two processing networks of the image processing model for processing to obtain corresponding processing results; wherein each processing network is constructed based on at least one preset task, and the processing network comprises a neck module and a detection module; the neck module is used to fuse image features to obtain fused features matching the preset task, and the detection module is used to detect the fused features to obtain processing results corresponding to the preset task. This method can avoid "parameter preemption" between different preset tasks, improve the robustness of the image processing model, and thereby improve the accuracy of the processing results of each preset task.

Description

Technical Field

[0001] The present application belongs to the field of image processing technology, and in particular relates to an image processing method, an image processing device, an electronic device, and a computer-readable storage medium. Background Art

[0002] In the related art, the network model for processing images can handle at least two tasks at the same time, such as a multi-task learning network model. Although such a network model can efficiently solve multiple tasks, the current design of network models mostly adopts the method of hard parameter sharing, specifically: the bottom-level parameters (parameters corresponding to the backbone network and the neck network) are shared, and the top-level parameters (parameters corresponding to the detection module) are independent of each other. This hard parameter sharing method will cause "parameter preemption" between different tasks, and "parameter preemption" will easily make the network model unable to take into account multiple tasks, which will lead to low robustness of the network model and reduce the accuracy of the processing results of each task. Summary of the invention

[0003] The present application provides an image processing method, an image processing device, an electronic device and a computer-readable storage medium, which can avoid "parameter preemption" between different tasks, improve the robustness of the image processing model, and further improve the accuracy of the processing results of each task.

[0004] In a first aspect, the present application provides an image processing method, comprising:

[0005] Based on the backbone network of the pre-trained image processing model, feature extraction is performed on the image to be processed to obtain image features;

[0006] The image features are respectively input into at least two processing networks of the image processing model for processing to obtain corresponding processing results; wherein each processing network is constructed based on at least one preset task, and the processing network includes a neck module and a detection module; the neck module is used to fuse image features to obtain fused features that match the preset task, and the detection module is used to detect the fused features to obtain the processing results corresponding to the preset task.

[0007] In a second aspect, the present application provides an image processing device, comprising:

[0008] An extraction module is used to extract features of the image to be processed based on the backbone network of the pre-trained image processing model to obtain image features;

[0009] A processing module is used to input image features into at least two processing networks of an image processing model for processing to obtain corresponding processing results; wherein each processing network is constructed based on at least one preset task, and the processing network includes a neck module and a detection module; the neck module is used to fuse image features to obtain fused features that match the preset task, and the detection module is used to detect the fused features to obtain a processing result corresponding to the preset task.

[0010] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method of the first aspect when executing the computer program.

[0011] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method of the first aspect are implemented.

[0012] In a fifth aspect, the present application provides a computer program product, which includes a computer program. When the computer program is executed by one or more processors, it implements the steps of the method of the first aspect.

[0013] Compared with the prior art, the present application has the following beneficial effects: the processing of the image to be processed is to complete at least two preset tasks. In order to avoid the situation of "parameter preemption" between different preset tasks and reduce the robustness of the image processing model, the present application constructs a processing network of the image processing model through preset tasks, and transforms the image processing model from "hard parameter sharing" to "soft parameter sharing", thereby obtaining an improved image processing model. Processing the image to be processed based on the improved image processing model can give full play to the learning ability of each processing network, so that each processing network can learn the unique features of the corresponding preset task in a targeted manner, thereby improving the accuracy of the processing results of the corresponding preset task.

[0014] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0016] Figure 1is a schematic diagram of the structure of the downsampling module provided in an embodiment of the present application;

[0017] Figure 2 It is a schematic diagram of the main structure of the feature extraction module provided in the embodiment of the present application;

[0018] Figure 3 is a schematic diagram of the structure of the CCT Block provided in an embodiment of the present application;

[0019] Figure 4 is a schematic diagram of the structure of the CFFN provided in the embodiment of the present application;

[0020] Figure 5 is a structural diagram of an image processing network model provided in an embodiment of the present application;

[0021] Figure 6 is a flowchart of an image processing method provided in an embodiment of the present application;

[0022] Figure 7 is a structural schematic diagram of an image processing device provided in an embodiment of the present application;

[0023] Figure 8 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0025] In the related technology, when the network model for processing images handles more than two tasks at the same time, "parameter preemption" occurs between different tasks because the network model is designed with hard parameter sharing. This "parameter preemption" easily makes it impossible for the network model to take into account multiple tasks, which in turn leads to low robustness of the network model and reduces the accuracy of the processing results of each task.

[0026] In order to solve this problem, the present application proposes an improved image processing model, which can avoid the "parameter preemption" between different preset tasks, improve the robustness of the image processing model, and thus improve the accuracy of the processing results of each task. The image processing model proposed in the present application will be described below through specific embodiments.

[0027] In some embodiments, the image processing model includes a backbone network and at least two processing networks, and the backbone network is connected to each processing network. Each processing network can be constructed according to at least one preset task. As an example only, a processing network can be constructed according to each preset task; or a processing network can be constructed according to at least one preset task of the same type. Such an image processing model, whose design is transformed from "hard parameter sharing" to "soft parameter sharing", can give full play to the learning ability of each processing network and learn the unique characteristics of the corresponding preset task in a targeted manner.

[0028] Specifically, the backbone network is used to extract features from the image to be processed, and the processing network is used to process the image features to obtain processing results corresponding to the preset tasks.

[0029] Each processing network includes a neck module and a detection model. The neck module is used to fuse the image features output by the backbone network to obtain fused features that match the preset task; the detection network is used to detect the fused features output by the neck network to obtain the processing results corresponding to the preset task.

[0030] In some embodiments, the processing network can be constructed by presetting the number of tasks.

[0031] Specifically, when all preset tasks include at least two processing tasks, a processing network can be constructed based on each processing task. That is, as many processing networks are constructed as there are processing tasks, and each processing network is independent of each other, that is, each processing network only learns the features corresponding to the corresponding processing task. For each processing task, it includes a neck module and a detection module that match the corresponding processing task.

[0032] Just as an example, if all preset tasks include pedestrian detection and vehicle detection, then for pedestrian detection, a pedestrian detection network can be constructed; for vehicle detection, a vehicle detection network can be constructed. The pedestrian detection network includes a neck module for obtaining pedestrian fusion features and a detection module for detecting pedestrians; the vehicle detection network includes a neck module for obtaining vehicle fusion features and a detection module for detecting vehicles.

[0033] Obviously, such an image processing model can process each processing task independently, so that there is no "parameter preemption" between the processing tasks, which can improve the robustness of the image processing model, and thus help improve the accuracy of the processing results of each processing task.

[0034] In some embodiments, considering that different tasks of the same type have strong correlations, the processing network can also be constructed by presetting the attribute of the task type, that is, the design concept of "hard parameter sharing" can be used between tasks of the same type.

[0035] Specifically, when all preset tasks include at least two types of processing tasks, and the number of processing tasks of each type is at least one, a processing network can be constructed for each type of processing task. That is, there are as many types of processing tasks as there are processing networks. Each processing network includes a neck module and at least one detection module that match the corresponding type of processing task; that is, for each processing task of the same type, a neck module is shared, but each processing task of the same type corresponds to a detection module. In other words, in the same type, no matter how many processing tasks there are, only one neck module of the corresponding processing network can be set, and one detection head is set for each processing task.

[0036] Based on the improvement of this structure, "hard parameter sharing" can be achieved between tasks with strong correlation, and "soft parameter sharing" can be achieved between tasks of different types. This can not only reduce the overfitting risk of a single processing task in each type, but also avoid the "parameter preemption" between processing tasks of different types, thereby improving the robustness of the image processing model.

[0037] Just as an example, assuming that there are two detection tasks and one segmentation task, a detection network can be constructed based on the two detection tasks, and the detection network includes a neck module and two detection modules; a segmentation network can be constructed based on a segmentation task, and the segmentation network includes a neck module and a detection head. That is, for detection tasks and segmentation tasks with weak task correlation, the "soft parameter sharing" mode can be adopted; for two specific detection tasks in the detection tasks with strong task correlation, the "hard parameter sharing" mode can be adopted to improve the robustness of the image processing model.

[0038] In some embodiments, in order to reduce the amount of calculation, the backbone network may include a downsampling module. Specifically, the downsampling module is used to perform a downsampling operation on the image to be processed to obtain basic features. It can be understood that the reason for performing the downsampling operation is to reduce the amount of calculation and facilitate subsequent feature extraction.

[0039] In the related art, most Transformer-based models use a large convolution to directly cut the image to be processed into non-overlapping image patches to achieve downsampling operations, but this approach will directly lose the 2D spatial features in the patch and many details and edge image information. In order to reduce the amount of calculation while retaining the details and edges of the image to be processed as much as possible, the downsampling module in this embodiment adopts a traditional convolution structure.

[0040] like Figure 1 As shown in the figure, the downsampling module is obtained by stacking two 3×3 convolutional layers. The convolution stride is 2. After each convolutional layer, there is a batch normalization (BN) layer and a Gaussian Error Linear Units (GELU) layer. Downsampling based on this downsampling module can retain image information such as details and edges in the image to be processed.

[0041] In some embodiments, in order to extract multi-scale features, the main structure of the feature extraction module adopts a stage-wise Transformer. That is, the backbone network also includes at least two stages of feature extraction modules, which are connected in series with the upsampling module, and the output of the upsampling module is the input of the feature extraction module of the first stage. The feature extraction module of each stage is used to extract features from the input features to obtain extraction results; wherein, the input features of the feature extraction module of the first stage are basic features; the input features of the feature extraction modules of other stages except the first stage are the extraction results obtained by the feature extraction modules of the previous stage; the extraction results of the feature extraction module of the last stage are image features.

[0042] Just as an example, Figure 2 As shown, it is assumed that there are three stages in total. For the first stage, its input feature is the basic feature output by the downsampling module. After processing, the extraction result of the first stage is obtained; for the second stage, its input feature is the extraction result of the first stage. After processing, the extraction result of the second stage is obtained; for the third stage, its input feature is the extraction result of the second stage. After processing, the extraction result of the third stage is obtained, where the extraction result of the third stage is also the image feature.

[0043] In some embodiments, the feature extraction module of each stage includes an embedding layer (PatchEmbedding) and a parallel extraction layer. Among them, Patch Embedding can increase the number of channels in each stage while obtaining the basic features after dimensionality reduction. The parallel extraction layer can take into account both local information and global information in the image and extract more comprehensive information.

[0044] In some embodiments, the success of CNN depends on its two inherent inductive biases, namely translation invariance and local correlation, so it usually performs better on small data sets than pure Transformer structures. However, due to the limited receptive field of CNN, it is difficult to capture global information, while the visual Transformer can capture the global information of a picture and determine long-distance dependencies, so the Transformer surpasses the CNN structure in many visual tasks. Based on this, in order to enable the network structure to inherit the advantages of CNN and Transformer, and to retain global and local features to the greatest extent, in this embodiment, CNN and Transformer are combined. Specifically, the parallel extraction layer is stacked by blocks (referred to as CCT Block) based on the fusion of convolutional CNN and Transformer features.

[0045] Each CCT Block includes local feature units (LFU) and global feature units (GFU), which are connected in parallel. LFU extracts local features from the basic features after dimensionality reduction to obtain local features, and GFU extracts global features from the basic features after dimensionality reduction to obtain global features. After fusing the global features and local features, they can be used as the output of the current CCT Block.

[0046] In some embodiments, the GFU includes a first subunit and a second subunit, the first subunit and the second subunit are connected in series, the first subunit includes a lightweight multi-head self-attention (LMHSA) module, and the second subunit includes a convolutional feed-forward network (CFFN).

[0047] See also Figure 3 , Figure 3The structural diagram of CCT Block is shown. In the diagram, LFU mainly includes 3×3 depthwise separable convolution (DWConv). GFU is mainly composed of LMHSA and CFFN. Among them, the first subunit is designed based on LMHSA, and it can be considered that the first subunit includes a first branch and a second branch, and the first branch and the second branch are connected in parallel; the first branch includes a layer normalization (LN) module and an LMHSA module, and the LN module and the LMHSA module are connected in series; the second branch is mainly used to transfer features. The second subunit is designed based on CFFN, and it can be considered that the second subunit includes a third branch and a fourth branch, and the third branch and the fourth branch are connected in parallel; the third branch includes an LN module and a CFFN network, and the LN module and the CFFN network are connected in series; the fourth branch is mainly used to transfer features. Finally, the output of LFU and the output of GFU are fused through Concat to output the fused features of a single CCT Block.

[0048] In some embodiments, given an input of size Rn×d, the original multi-head attention mechanism first generates the corresponding Query, Key and Value (consistent with the original input size), and then generates a weight matrix of size Rn×n by dot product of Query and Key. However, this process often consumes a lot of computing resources (video memory) due to the large input feature size, which makes it difficult to train and deploy the network. Based on this, this embodiment adopts LMHSA, and two k×k depth-separable convolutions can be used to downsample the generation of Key and Value respectively, to obtain two relatively small features K' and V', which can be expressed as follows:

[0049]

[0050]

[0051]

[0052] MultiHead(Q,K',V')=Concat(head 1 ,...,head h )W O

[0053] In some embodiments, see Figure 4 , Figure 4A schematic diagram of the structure of CFFN is shown. Compared with the Transformer's Feed Forward Networks (FFN), CFFN adds DWConv between two fully connected layers, which can enhance the locality of the Transformer while ensuring information exchange between windows.

[0054] In some embodiments, based on the specific implementation schemes in the above embodiments, it is assumed that the preset tasks are two detection tasks and one segmentation task, so there are two processing networks, namely the detection network and the segmentation network. The corresponding image processing network model is as follows: Figure 5 As shown. Figure 5 In the embodiment, CCTstem is the down-sampling module in the above embodiment. For the specific structure, please refer to Figure 2 There are four stages in total. The Patch Embed in each stage is the Patch Embedding in the aforementioned embodiment. The CCT Block structure in each stage can be found in the aforementioned Figure 3 and Figure 4 ; For the image features output in the fourth stage, they can be respectively input into the neck module of the detection network and the neck module of the segmentation network via spatial pyramid pooling (SPP). In order to allow the neck networks of the two processing networks to obtain features at different levels and better characterize the features of different targets, for each neck network, the output results of the second and third stages can also be fused. For the detection network, two detection modules are set up, and the fused features of the output of the neck module of the detection network can be respectively input into the two detection modules to obtain the detection results corresponding to different detection tasks. For the segmentation network, only one segmentation module is set up, and the fused features of the output of the neck module of the detection network can be input into the segmentation module to obtain the corresponding segmentation result.

[0055] Based on the image processing model corresponding to the above embodiment, the present application provides an image processing method.

[0056] The image processing method provided in the embodiments of the present application can be applied to electronic devices such as mobile phones, tablet computers, vehicle-mounted equipment, augmented reality (AR) / virtual reality (VR) equipment, laptop computers, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), etc. The embodiments of the present application do not impose any restrictions on the specific types of electronic devices.

[0057] Figure 6 A schematic flow chart of an image processing method provided by the present application is shown, and the image processing method includes:

[0058] Step 610: Extract features of the image to be processed based on the backbone network of the pre-trained image processing model to obtain image features.

[0059] Step 620: Input the image features into at least two processing networks of the image processing model for processing to obtain corresponding processing results.

[0060] Among them, each processing network is constructed based on at least one preset task, and the processing network includes a neck module and a detection module; the neck module is used to fuse image features to obtain fused features matching the preset task, and the detection module is used to detect the fused features to obtain the processing results corresponding to the preset task.

[0061] Since each step of the image processing method in this embodiment has been introduced in the embodiment of the aforementioned image processing model, it will not be repeated in this embodiment for the sake of brevity; it can be understood that the same applies to the other embodiments.

[0062] In some embodiments, all preset tasks are at least two processing tasks, each processing task corresponds to a processing network, and each processing network includes a neck module and a detection module that match the corresponding processing task.

[0063] In some embodiments, all preset tasks are at least two types of processing tasks, and the number of each type of processing task is at least one; one processing network corresponds to one type of processing task, and each processing network includes a neck module and at least one detection module that match the corresponding type of processing task.

[0064] In some embodiments, the backbone network includes a downsampling module and at least two stages of feature extraction modules. The backbone network based on the pre-trained image processing model extracts features from the image input to be processed. The aforementioned step 610 specifically includes:

[0065] Step 611: Perform a downsampling operation on the image to be processed based on the downsampling module to obtain basic features.

[0066] Step 612: For the feature extraction module of each stage, feature extraction is performed on the input features based on the feature extraction module to obtain extraction results; wherein, the input features of the feature extraction module of the first stage are basic features; the input features of the feature extraction modules of other stages except the first stage are the extraction results obtained by the feature extraction modules of the previous stage; and the extraction results of the feature extraction module of the last stage are image features.

[0067] In some embodiments, the feature extraction module includes an embedding layer and a parallel extraction layer, and step 612 specifically includes:

[0068] Step 6121: input the basic features into the embedding layer for dimensionality reduction processing to obtain the basic features after dimensionality reduction.

[0069] Step 6122: Input the basic features after dimensionality reduction into the parallel extraction layer for feature extraction to obtain the extraction results.

[0070] In some embodiments, the parallel extraction layer includes a local feature extraction unit and a global feature extraction unit, and the local feature extraction unit and the global feature extraction unit are connected in parallel. The above step 6121 specifically includes:

[0071] Step A1: Input the basic features after dimension reduction into the local feature extraction unit to obtain local features.

[0072] Step A2: Input the basic features after dimension reduction into the global feature extraction unit to obtain the global features.

[0073] Step A3: Fuse local features and global features to obtain extraction results.

[0074] In some embodiments, the global feature extraction unit includes a first subunit and a second subunit, the first subunit and the second subunit are connected in series, the first subunit includes a lightweight multi-head attention module, and the second subunit includes a convolutional feedforward network; the above step A2 specifically includes:

[0075] Step A21: Input the basic features after dimensionality reduction into the first subunit for feature extraction to obtain initial global features.

[0076] Step A22: input the initial global feature into the second subunit for feature extraction to obtain the global feature.

[0077] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0078] In some application scenarios, the above-mentioned image processing model can be a multi-task vehicle-mounted panoramic driving perception model. The multi-tasks include traffic sign detection tasks, drivable area segmentation tasks, lane line segmentation tasks, obstacle detection tasks, and pedestrian detection tasks. Among them, it can be determined to set at least one detection network based on the correlation between the three detection tasks. For example, when it is determined that the three detection tasks are strongly correlated, only one detection network can be set, and the detection network includes a neck module and three detection modules. Or when it is determined that two of the detection tasks are strongly correlated, for the two strongly correlated detection tasks, a detection network is set, and the detection network includes a neck module and two detection modules; for the remaining detection network, another detection network is set separately, and the detection network includes a neck module and a detection module. For the setting of the segmentation network, the setting of the detection network is analogous, and will not be repeated.

[0079] Based on the above settings, "hard parameter sharing" is achieved between tasks with strong correlation, and "soft parameter sharing" is achieved between tasks of different types. This can not only reduce the overfitting risk of a single processing task in each type, but also avoid the "parameter preemption" between processing tasks of different types, thereby improving the robustness of the driving perception model.

[0080] The backbone network of the driving perception model is a multi-task learning network based on the fusion of Conv and Transformer features. This backbone network not only inherits the advantages of CNN and Transformer, but also gradually fuses the extraction results of CNN and Transformer at the block level, which can retain multi-scale global and local features to the greatest extent, and can more accurately locate and identify targets; it helps to improve the stability and reliability of vehicle-mounted panoramic driving perception.

[0081] In some embodiments, for a constructed driving perception model, a corresponding data set can be obtained based on each task, and different data sets can be processed differently to obtain better training samples. The driving perception model is trained with training samples, and when the driving perception model converges, the training can be stopped to obtain a trained driving perception model. Processing various detection tasks based on the trained driving perception model can obtain processing results with higher accuracy, which helps to improve the stability and reliability of intelligent driving.

[0082] Optionally, for the traffic sign dataset, the annotation method adopts rectangular detection box annotation, which can be in xml format; for the drivable area segmentation dataset, the annotation method adopts connected domain annotation method, which can be in json format; for the lane line segmentation, the annotation method adopts connected domain annotation method, which can be in json format.

[0083] After the datasets are labeled, in order to improve the generalization ability of the driving perception model, data enhancement can be performed on each image in each dataset to increase the complexity of the image. Data enhancement may include random scaling, random cropping, random arrangement, random rotation, and splicing.

[0084] Corresponding to the image processing method of the above embodiment, Figure 7 A structural block diagram of an image processing device 7 provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown.

[0085] Reference Figure 7 , the image processing device 7 comprises:

[0086] The extraction module 71 is used to extract features of the image to be processed based on the backbone network of the pre-trained image processing model to obtain image features.

[0087] The processing module 72 is used to input the image features into at least two processing networks of the image processing model for processing to obtain corresponding processing results; wherein each processing network is constructed based on at least one preset task, and the processing network includes a neck module and a detection module; the neck module is used to fuse the image features to obtain a fused feature that matches the preset task, and the detection module is used to detect the fused feature to obtain a processing result corresponding to the preset task.

[0088] Optionally, the backbone network includes a downsampling module and at least two stages of feature extraction modules. The backbone network based on the pre-trained image processing model performs feature extraction on the image input to be processed. The extraction module 71 may include:

[0089] The downsampling unit is used to perform a downsampling operation on the image to be processed based on the downsampling module to obtain basic features.

[0090] The stage extraction unit is used to extract features from the input features of the feature extraction module of each stage based on the feature extraction module to obtain extraction results; wherein, the input features of the feature extraction module of the first stage are basic features; the input features of the feature extraction modules of other stages except the first stage are the extraction results obtained by the feature extraction modules of the previous stage; the extraction results of the feature extraction module of the last stage are image features.

[0091] In some embodiments, the feature extraction module includes an embedding layer and a parallel extraction layer, and the stage extraction unit may include:

[0092] The first extraction subunit is used to input the basic features after dimension reduction into the local feature extraction unit to obtain local features.

[0093] The second extraction subunit is used to input the basic features after dimension reduction into the global feature extraction unit to obtain the global features.

[0094] The fusion subunit is used to fuse local features with global features to obtain the extraction results.

[0095] Optionally, the global feature extraction unit includes a first subunit and a second subunit, the first subunit and the second subunit are connected in series, the first subunit includes a lightweight multi-head attention module, and the second subunit includes a convolutional feedforward network; the second extraction subunit includes:

[0096] The first extraction grandchild unit is used to input the basic features after dimensionality reduction into the first subunit for feature extraction to obtain the initial global features.

[0097] The second extraction subunit is used to input the initial global features into the second subunit for feature extraction to obtain global features.

[0098] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0099] Figure 8 This is a schematic diagram of the physical structure of an electronic device provided by an embodiment of the present application. Figure 8 As shown, the electronic device 8 of this embodiment includes: at least one processor 80 ( Figure 8 Only one is shown in the figure) a processor, a memory 81, and a computer program 82 stored in the memory 81 and executable on at least one processor 80. When the processor 80 executes the computer program 82, the steps in any of the above-mentioned image processing method embodiments are implemented, for example Figure 6 Steps 610-620 are shown.

[0100] The processor 80 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0101] In some embodiments, the memory 81 may be an internal storage unit of the electronic device 8, such as a hard disk or memory of the electronic device 8. In other embodiments, the memory 81 may also be an external storage device of the electronic device 8, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 8.

[0102] Furthermore, the memory 81 may also include both an internal storage unit of the electronic device 8 and an external storage device. The memory 81 is used to store operating means, application programs, boot loaders, data and other programs, such as program codes of computer programs, etc. The memory 81 may also be used to temporarily store data that has been output or is to be output.

[0103] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the above-mentioned device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0104] An embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments can be implemented.

[0105] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0106] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program. The above-mentioned computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the above-mentioned computer program includes computer program code, and the above-mentioned computer program code can be in source code form, object code form, executable file or some intermediate form. The above-mentioned computer-readable medium may at least include: any entity or device that can carry the computer program code to the camera / electronic device, recording medium, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium. For example, a USB flash drive, a mobile hard disk, a disk or an optical disk.

[0107] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0108] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0109] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the above modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0110] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0111] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for processing an image, characterized in that: include: Based on the backbone network of the pre-trained image processing model, feature extraction is performed on the image to be processed to obtain image features; The image features are respectively input into at least two processing networks of the image processing model for processing to obtain corresponding processing results; wherein each of the processing networks is constructed based on at least one preset task, and the processing network includes a neck module and a detection module; the neck module is used to fuse the image features to obtain a fusion feature matching the preset task, and the detection module is used to detect the fusion feature to obtain the processing result corresponding to the preset task; the image processing model is used to process at least two preset tasks at the same time, and the parameter sharing mode between the preset tasks corresponding to each of the processing networks is soft parameter sharing; Wherein, all the preset tasks include at least two processing tasks, each of the processing tasks corresponds to one of the processing networks, and each of the processing networks includes one of the neck modules and one of the detection modules matched with the corresponding processing tasks; Alternatively, all of the preset tasks include at least two types of processing tasks, and the number of each type of processing task is at least one; one processing network corresponds to one type of processing task, and each processing network includes one neck module and at least one detection module that match the corresponding type of processing task.

2. The image processing method according to claim 1, characterized in that: The backbone network includes a downsampling module and at least two stages of feature extraction modules. The backbone network based on the pre-trained image processing model extracts features from the image input to be processed, including: Performing a downsampling operation on the image to be processed based on the downsampling module to obtain basic features; For the feature extraction module in each stage, feature extraction is performed on the input features based on the feature extraction module to obtain an extraction result; wherein, the input features of the feature extraction module in the first stage are the basic features; the input features of the feature extraction modules in other stages except the first stage are the extraction results obtained by the feature extraction modules in the previous stage; and the extraction results of the feature extraction module in the last stage are the image features.

3. The image processing method according to claim 2, characterized in that: The feature extraction module includes an embedding layer and a parallel extraction layer, and the feature extraction of the input features based on the feature extraction module includes: Inputting the basic features into the embedding layer for dimensionality reduction processing to obtain the basic features after dimensionality reduction; The basic features after dimension reduction are input into the parallel extraction layer to perform feature extraction to obtain the extraction results.

4. The image processing method according to claim 3, characterized in that: The parallel extraction layer includes a local feature extraction unit and a global feature extraction unit, the local feature extraction unit and the global feature extraction unit are connected in parallel, and the basic features are input into the embedding layer for dimensionality reduction processing, including: Inputting the basic features after dimension reduction into the local feature extraction unit to obtain local features; Inputting the basic features after dimension reduction into the global feature extraction unit to obtain global features; The local features and the global features are fused to obtain the extraction result.

5. The image processing method according to claim 4, characterized in that: The global feature extraction unit includes a first subunit and a second subunit, the first subunit and the second subunit are connected in series, the first subunit includes a lightweight multi-head attention module, and the second subunit includes a convolutional feedforward network; the inputting the basic features after dimensionality reduction into the global feature extraction unit includes: Inputting the basic features after dimension reduction into the first subunit for feature extraction to obtain initial global features; The initial global feature is input into the second subunit to perform feature extraction to obtain the global feature.

6. An image processing device, characterized in that: include: An extraction module is used to extract features of the image to be processed based on the backbone network of the pre-trained image processing model to obtain image features; A processing module, used for inputting the image features into at least two processing networks of the image processing model for processing respectively, and obtaining corresponding processing results; wherein each of the processing networks is constructed based on at least one preset task, and the processing network includes a neck module and a detection module; the neck module is used for fusing the image features to obtain a fusion feature matching the preset task, and the detection module is used for detecting the fusion feature to obtain the processing result corresponding to the preset task; the image processing model is used for processing at least two preset tasks at the same time, and the parameter sharing mode between the preset tasks corresponding to each of the processing networks is soft parameter sharing; Wherein, if all the preset tasks include at least two processing tasks, each of the processing tasks corresponds to one of the processing networks, and each of the processing networks includes one of the neck modules and one of the detection modules matched with the corresponding processing tasks; Alternatively, if all the preset tasks include at least two types of processing tasks, the number of each type of processing tasks is at least one; one processing network corresponds to one type of processing task, and each processing network includes one neck module and at least one detection module that match the corresponding type of processing task.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the image processing method according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the image processing method according to any one of claims 1 to 5 is implemented.

Images