Training method of image processing model, image processing method, electronic equipment and medium

Abstract

The invention provides an image processing model training method, an image processing method, electronic equipment and a medium, and the method comprises the steps: receiving an image sample and a corresponding tag, and the tag comprises a target positioning tag and an attribute classification tag; inputting the image sample and the label into a pre-constructed image processing model, wherein a prediction head of the pre-constructed image processing model comprises a prediction branch for predicting target positioning and a plurality of parallel prediction branches for predicting attributes; and training the pre-constructed image processing model according to the target positioning tag, the attribute classification tag, the prediction branch for predicting target positioning and the plurality of parallel prediction branches for predicting attributes to obtain a trained image processing model.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a training method for an image processing model, an image processing method, an electronic device and a medium. Background Art

[0002] In the process of image processing and recognition, the location and attribute labels of target features can be output according to the input image. In related technologies, target location and attribute classification in the image recognition process are implemented by two models. First, the target detection model is used to locate the position of the target object in the image, and then the target area is cropped and input into the attribute classification model to perform single-task or multi-task attribute classification. This method is costly and has low accuracy. Summary of the invention

[0003] The present application provides an image processing model training method, an image processing method, an electronic device and a medium, which can reduce the cost of image recognition.

[0004] The first aspect of the present application discloses a training method for an image processing model for use in an electronic device, the method comprising: receiving an image sample and a corresponding label, the label comprising a target location label and an attribute classification label; inputting the image sample and the label into a pre-constructed image processing model, wherein the prediction head of the pre-constructed image processing model comprises a prediction branch for predicting target location and a plurality of parallel prediction branches for predicting attributes; training the pre-constructed image processing model according to the target location label, the attribute classification label, the prediction branch for predicting target location, and the plurality of parallel prediction branches for predicting attributes to obtain a trained image processing model.

[0005] In a possible implementation of the first aspect above, the prediction branch is a convolutional layer.

[0006] In a possible implementation of the first aspect above, the prediction branch is a fully connected layer.

[0007] In a possible implementation of the first aspect above, the image samples include positive samples and negative samples, wherein labels of the positive samples are obtained by manual annotation.

[0008] In a possible implementation of the first aspect above, the method also includes training the pre-constructed image processing model according to the target positioning label, the attribute classification label, the prediction branch for predicting target positioning, and the multiple parallel prediction branches for predicting attributes, including first training according to the prediction branch for predicting target positioning and the target positioning label, and then training according to the prediction branch for predicting attributes and the attribute classification label.

[0009] In a possible implementation of the first aspect above, the pre-built image processing model includes at least one of single-stage SSD, dual-stage faster-rcnn, mask-rcnn and YoloV5.

[0010] In a possible implementation of the first aspect above, the target location tag includes a position of the target in the image.

[0011] In a possible implementation of the first aspect above, the attribute classification label includes at least one of the gender, age, type of the target, and the area where the target is displayed.

[0012] The second aspect of the present application discloses an image processing method, which includes receiving an image; inputting the image into an image processing model to output the location of the target in the image and the attribute classification of the target, wherein the image processing model is trained according to the method of the first aspect of the present application.

[0013] The third aspect of the present application discloses an electronic device, which includes a memory storing computer-executable instructions and a processor; when the instructions are executed by the processor, the device implements the methods according to the first and second aspects of the present application.

[0014] The fourth aspect of the present application discloses a computer-readable medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the methods described in the first and second aspects of the present application.

[0015] The fifth aspect of the present application discloses a computer program product, which implements the methods of the first and second aspects of the present application when the computer program is executed by a processor.

[0016] The training method of the image processing model, the image processing method, the electronic device and the medium provided by the present application can output the target location and multiple attribute classification information in parallel by inputting the image sample into a pre-built image processing model including parallel prediction branches. The model is then trained according to the difference between the training value output by the model and the label of the sample, and finally the trained image processing model is obtained. Method 200 realizes multi-task attribute classification based on the target detection model, which can reduce costs and improve image processing accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram of a scene in image recognition;

[0018] Figure 2 A flowchart of a method 200 for training an image processing model according to an embodiment of the present application;

[0019] Figure 3 A schematic diagram of the structure of a pre-built image processing model 300 according to an embodiment of the present application;

[0020] Figure 4 A flowchart of an image processing method 400 according to an embodiment of the present application;

[0021] Figure 5 FIG. 5 is a block diagram of an electronic device 500 according to an embodiment of the present application. DETAILED DESCRIPTION

[0022] The present application is further described below in conjunction with specific embodiments and drawings. It is understood that the illustrative embodiments of the present disclosure include, but are not limited to, training methods of image processing models, image processing methods, devices, electronic devices, and media, and the specific embodiments described herein are only for the purpose of explaining the present application, rather than limiting the present application. In addition, for ease of description, only some, but not all, structures or processes related to the present application are shown in the drawings.

[0023] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0024] Figure 1 A schematic diagram of a scene in image recognition is shown. The picture 101 is input into the image processing and recognition process 102, and the location of the feature and multiple related attribute labels can be output. Usually, the image segmentation technology based on the target geometry and statistical features can be used to detect the object object in the image and the location of the object object in the image. The object object can be a person, an animal, or an object. The attribute label can be the gender or age of the person, the breed of the animal, the classification of the object, or the area where the feature is displayed, such as the head or the bust. It can also be a close-up or a distant view.

[0025] In related technologies, target positioning and attribute classification in the image recognition process are implemented by two models. First, the target detection model is used to locate the position of the target object in the image, and then the target area is cropped and input into the attribute classification model to perform single-task or multi-task attribute classification. On the one hand, this method is not conducive to cost control because the costs of calculation, storage and communication are relatively high; on the other hand, the input of the attribute classification model is only the sub-area extracted from the original image, so some context information will be lost, thereby reducing accuracy.

[0026] In order to solve the above problems, an embodiment of the present application provides a training method 200 for an image processing model. Figure 2 . Figure 2 The steps are as follows.

[0027] In S210, an image sample and a corresponding label are received, where the label includes a target location label and an attribute classification label.

[0028] Image samples and corresponding labels can be obtained from the labeled image data. The label corresponding to the image includes not only the location of the object target in the image, but also the attribute classification corresponding to the object.

[0029] In some embodiments, the target location tag includes the location of the target in the image, and the attribute classification tag includes at least one of the target's gender, age, type, and the region where the target is displayed.

[0030] In some embodiments, the image samples include positive samples and negative samples, wherein only the labels of the positive samples are obtained by manual annotation, that is, only the positive samples contain the ground-truth of the attribute class and participate in the gradient backpropagation training.

[0031] At S220 , the image sample and the label are input into a pre-built image processing model, wherein the prediction head of the pre-built image processing model includes a prediction branch for predicting target location and a plurality of parallel prediction branches for predicting attributes.

[0032] The image samples and corresponding labels are input into a pre-built image processing model. In some embodiments, the pre-built image processing model can be built based on a single-stage SSD, Yolo or a two-stage faster-rcnn, mask-rcnn model. The single-stage target detection method generally refers to the target detection that can be achieved by extracting features once. The two-stage target detection method first finds the position of the target object in the first stage and obtains the suggestion box; in the second stage, the suggestion box is classified to find a more accurate position.

[0033] In the prediction head part of the pre-built image processing model, there are multiple parallel prediction branches, including a prediction branch for predicting target location and a prediction branch for predicting multiple attributes. The prediction branches for predicting multiple attributes can be used to predict multiple attribute values ​​respectively.

[0034] In S230, the pre-built image processing model is trained according to the output of the prediction branch and the target location label and the attribute classification label to obtain a trained image processing model.

[0035] The pre-built image processing model can be trained based on the output of the prediction branch, that is, the training value of the target positioning and the training value of the attribute classification label, by the difference between the training value and the label, and finally obtain the trained image processing model.

[0036] Figure 2The training method 200 of the image processing model in the invention can output the target location and multiple attribute classification information in parallel by inputting the image sample into the target detection model including the parallel prediction branch. Then the model is trained according to the difference between the training value output by the model and the label of the sample, and finally the trained image processing model is obtained. The method 200 realizes multi-task attribute classification based on the target detection model, which can reduce the cost and improve the image processing accuracy.

[0037] In some embodiments, the prediction branch of the pre-built image processing model may be a convolutional layer. Figure 3 A schematic diagram of the structure of a pre-built image processing model 300 according to an embodiment of the present application is shown.

[0038] The landmark detection model 300 is based on Yolov5 as an example, and includes an input terminal (not shown), a backbone network (Backbone) 301, a neck (Neck) 302, and a prediction output terminal 303. Backbone 301 is a convolutional neural network that aggregates and forms image features at different image granularities for feature extraction. Neck 302 is a series of network layers that mix and combine image features and pass the image features to the prediction output terminal 303.

[0039] Figure 3 The structure of Yolov5 in the above method is different from that of conventional Yolov5 in that multiple prediction branch structures 304 are added to the prediction output terminal 303, i.e., in the model prediction head part. Figure 3 In the example, the parallel prediction branch is a 1x1 convolutional (Conv) layer.

[0040] The training method of the image processing model of the present application has a wide range of applications. Figure 3 The model 300 is not limited to application to Yolov5, but can also be applied to other target detection models, such as ssd, Yolo, faster-rcnn, or mask-rcnn.

[0041] In some implementations, the prediction branch in the pre-built image processing model may be a fully connected layer.

[0042] In some embodiments, during the training of the image processing model, the pre-built image processing model is trained according to the target location label, the attribute classification label, the prediction branch for predicting the target location, and the multiple parallel prediction branches for predicting the attributes, including first training according to the prediction branch for predicting the target location and the target location label, and then training according to the prediction branch for predicting the attributes and the attribute classification label. This method of first training the location and classification loss, and then performing the attribute prediction loss, makes the training tasks from easy to difficult, thereby improving the convergence speed and stability of the model.

[0043] Figure 4 An image processing method 400 according to an embodiment of the present application is shown, comprising the following steps.

[0044] In S410, an image is received.

[0045] In S420, the image is input into the image processing model to output the location of the object in the image and the attribute classification of the object.

[0046] The image processing model can be applied by Figure 2 The training method is obtained. Figure 4 When processing an image, method 400 may output the location of the target, and a single or multiple attribute classifications.

[0047] Reference now Figure 5 , which is a block diagram of an electronic device 500 according to an embodiment of the present application. The electronic device 500 may include one or more processors 502, a system control logic 508 connected to at least one of the processors 502, a system memory 504 connected to the system control logic 508, a non-volatile memory (NVM) 506 connected to the system control logic 508, and a network interface 510 connected to the system control logic 508.

[0048] The processor 502 may include one or more single-core or multi-core processors. The processor 502 may include any combination of general-purpose processors and special-purpose processors (e.g., graphics processors, application processors, baseband processors, etc.). In the embodiments of the present invention, the processor 502 may be configured to execute the following operations: Figure 2-4 One or more of the various embodiments shown.

[0049] In some embodiments, system control logic 508 may include any suitable interface controller to provide any suitable interface to at least one of processors 502 and / or any suitable device or component in communication with system control logic 508 .

[0050] In some embodiments, the system control logic 508 may include one or more memory controllers to provide an interface to the system memory 504. The system memory 504 may be used to load and store data and / or instructions. In some embodiments, the memory 504 of the device 500 may include any suitable volatile memory, such as a suitable dynamic random access memory (DRAM).

[0051] NVM / memory 506 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. In some embodiments, NVM / memory 506 may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device, such as at least one of a HDD (Hard Disk Drive), a CD (Compact Disc) drive, and a DVD (Digital Versatile Disc) drive.

[0052] NVM / storage 506 may include a portion of storage resources installed on a device of apparatus 500, or it may be accessible to the apparatus but not necessarily a part of the apparatus. For example, NVM / storage 506 may be accessed over a network via network interface 510.

[0053] In particular, system memory 504 and NVM / storage 506 may include, respectively, a temporary copy and a permanent copy of instructions 520. Instructions 520 may include instructions that, when executed by at least one of processors 502, cause device 500 to perform the following operations: Figure 2 , 4 In some embodiments, instructions 520 , hardware, firmware, and / or software components thereof may additionally / alternatively be placed in system control logic 508 , network interface 510 , and / or processor 502 .

[0054] The network interface 510 may include a transceiver for providing a radio interface for the device 500, thereby communicating with any other suitable device (such as a front-end module, an antenna, etc.) through one or more networks. In some embodiments, the network interface 510 may be integrated with other components of the device 500. For example, the network interface 510 may be integrated with at least one of the processor 502, the system memory 504, the NVM / storage 506, and a firmware device (not shown) having instructions. When at least one of the processors 502 executes the instructions, the device 500 implements Figure 2-4 One or more of the various embodiments shown.

[0055] The network interface 510 may further include any suitable hardware and / or firmware to provide a multiple-input multiple-output radio interface. For example, the network interface 510 may be a network adapter, a wireless network adapter, a telephone modem and / or a wireless modem.

[0056] In one embodiment, at least one of the processors 502 may be packaged together with logic for one or more controllers of the system control logic 508 to form a system in package (SiP). In one embodiment, at least one of the processors 502 may be integrated on the same die with logic for one or more controllers of the system control logic 508 to form a system on a chip (SoC).

[0057] The device 500 may further include an input / output (I / O) device 512. The I / O device 512 may include a user interface, which enables a user to interact with the device 500; and a peripheral component interface design enables peripheral components to interact with the device 500.

[0058] In some embodiments, the user interface may include, but is not limited to, a display (e.g., an LCD display, a touch screen display, etc.), a speaker, a microphone, one or more cameras (e.g., a still image camera and / or a video camera), a flashlight (e.g., an LED flash), and a keyboard.

[0059] In some embodiments, the peripheral component interface may include, but is not limited to, a non-volatile memory port, an audio jack, and a power interface.

[0060] Each method implementation mode of the present application can be implemented in the form of software, magnetic components, firmware, etc.

[0061] Program code can be applied to input instructions to perform the functions described herein and generate output information. The output information can be applied to one or more output devices in a known manner. For purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0062] Program code can be implemented with high-level programming language or object-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described herein is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0063] One or more aspects of at least one embodiment may be implemented by representative instructions stored on a computer-readable storage medium, which represent various logic in a processor, and when the instructions are read by a machine, cause the machine to fabricate logic for performing the techniques described herein. These representations, known as "IP cores," may be stored on a tangible computer-readable storage medium and supplied to multiple customers or manufacturing facilities to load into a manufacturing machine that actually manufactures the logic or processor.

[0064] An embodiment of the present application discloses a computer-readable medium storing one or more programs executable by one or more processors to implement the method of the present application.

[0065] An embodiment of the present application discloses a computer program product, including a computer program, which implements the method of the present application when executed by a processor.

[0066] The above is a description of the implementation mode of the present application by specific specific embodiments. Those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Although the description of the present application will be introduced in conjunction with the preferred embodiment, this does not mean that the features of this invention are limited to this implementation mode. On the contrary, the purpose of introducing the invention in conjunction with the implementation mode is to cover other options or modifications that may be extended based on the claims of the present application. In order to provide an in-depth understanding of the present application, the following description will include many specific details. The present application can also be implemented without using these details. In addition, in order to avoid confusion or blurring the focus of the present application, some specific details will be omitted in the description. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0067] Furthermore, various operations will be described as multiple discrete operations in a manner that is most helpful for understanding the illustrative embodiments; however, the order of description should not be construed as implying that these operations are necessarily order dependent. In particular, these operations do not need to be performed in the order presented.

[0068] Unless the context dictates otherwise, the terms "comprising," "having," and "including" are synonymous. The phrase "A / B" means "A or B." The phrase "A and / or B" means "(A and B) or (A or B)."

[0069] As used herein, the term "module" or "unit" may refer to, be or include: an application specific integrated circuit (ASIC), an electronic circuit, a (shared, dedicated or group) processor and / or memory that executes one or more software or firmware programs, a combinational logic circuit and / or other suitable components that provide the described functionality.

[0070] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, instructions may be distributed over a network or by other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, but not limited to, floppy disks, optical disks, optical disks, read-only memories (CD-ROMs), magneto-optical disks, read-only memories (ROMs), random access memories (RAMs), erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), magnetic or optical cards, flash memory, or a tangible machine-readable memory for transmitting information through the Internet via electrical, optical, acoustic, or other forms of propagation signals (e.g., carrier waves, infrared signals, digital signals, etc.). Therefore, machine-readable media include any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a machine (e.g., computer) readable form.

[0071] In the accompanying drawings, some structural or method features are shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be required. In some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of structural or method features in a specific figure does not mean that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

[0072] It should be understood that although the terms "first", "second", etc. may be used here to describe various units or data, these units or data should not be limited by these terms. These terms are used only to distinguish one feature from another. For example, without departing from the scope of the exemplary embodiment, a first feature may be referred to as a second feature, and similarly, a second feature may be referred to as a first feature.

[0073] It should be noted that in this specification, similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0074] While the present invention has been shown and described with reference to certain preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention.

Claims

1. A training method for an image processing model, used in electronic equipment, characterized in that: The method comprises, Receiving image samples and corresponding labels, wherein the labels include target location labels and attribute classification labels; Inputting the image sample and the label into a pre-built image processing model, wherein the prediction head of the pre-built image processing model includes a prediction branch for predicting target location and a plurality of parallel prediction branches for predicting attributes; The pre-constructed image processing model is trained according to the target location label, the attribute classification label, the prediction branch for predicting target location, and the multiple parallel prediction branches for predicting attributes to obtain a trained image processing model.

2. The method according to claim 1, characterized in that The prediction branch is a convolutional layer.

3. The method according to claim 1, characterized in that The prediction branch is a fully connected layer.

4. The method according to claim 1, characterized in that: The image samples include positive samples and negative samples, wherein the labels of the positive samples are obtained by manual annotation.

5. The method according to claim 1, characterized in that Training the pre-constructed image processing model according to the target location label, the attribute classification label, the prediction branch for predicting target location, and the multiple parallel prediction branches for predicting attributes includes first training according to the prediction branch for predicting target location and the target location label, and then training according to the prediction branch for predicting attributes and the attribute classification label.

6. The method according to claim 1, characterized in that The pre-built image processing model includes at least one of single-stage SSD, dual-stage faster-rcnn, mask-rcnn and YoloV5.

7. The method according to claim 1, characterized in that The object location tag includes the location of the object in the image.

8. The method according to claim 1, characterized in that The attribute classification label includes at least one of the gender, age, type of the target, and the area where the target is displayed.

9. An image processing method, characterized in that: The method comprises, receiving an image; The image is input into an image processing model to output the location of the target in the image and the attribute classification of the target, wherein the image processing model is trained according to the method of claims 1-8.

10. An electronic device, characterized in that: The device comprises a memory storing computer executable instructions and a processor; when the instructions are executed by the processor, the device implements the method according to any one of claims 1 to 9.

11. A computer readable medium, characterized in that The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method according to any one of claims 1 to 9.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images