Image processing method and device, electronic equipment and computer readable storage medium

Abstract

The application discloses an image processing method and device, an electronic device and a computer readable storage medium. The method comprises the following steps: acquiring an image to be processed and a target detection network, wherein the target detection network is trained by a first training image, a second training image and at least one first interference image, the at least one first interference image is obtained by performing image interference processing on the first training image, first annotation data of the first training image is a first category of the first training image, second annotation data of the second training image comprises a second category of a detection object in the second training image and a first position of the detection object in the second training image; and performing target detection on the image to be processed by using the target detection network to obtain a first detection result.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to an image processing method and apparatus, electronic device and computer-readable storage medium. Background Technology

[0002] Thanks to the rapid development of deep learning technology, the performance of deep learning networks has become increasingly powerful. Therefore, deep learning networks are widely used in image processing, including object detection. However, current deep learning networks used for object detection have poor robustness, leading to low accuracy in object detection results. Summary of the Invention

[0003] This application provides an image processing method and apparatus, an electronic device, and a computer-readable storage medium.

[0004] This application provides an image processing method, the method comprising:

[0005] The image to be processed and the target detection network are obtained. The target detection network is trained by a first training image, a second training image and at least one first interference image. The at least one first interference image is obtained by performing image interference processing on the first training image. The first annotation data of the first training image is the first category of the first training image. The second annotation data of the second training image includes the second category of the detected object in the second training image and the first position of the detected object in the second training image.

[0006] The target detection network is used to perform target detection on the image to be processed to obtain a first detection result.

[0007] In conjunction with any embodiment of this application, the acquisition of the target detection network includes:

[0008] Obtain the network to be trained, the first training image, the second training image, and the at least one first interference image;

[0009] The first training image and the at least one first interference image are processed using the network to be trained to obtain a robust loss;

[0010] The target detection loss is obtained by processing the second training image using the network to be trained.

[0011] Based on the target detection loss and the robust loss, the total loss of the network to be trained is obtained;

[0012] Based on the total loss, the parameters of the network to be trained are updated to obtain the target detection network.

[0013] In any embodiment of this application, the step of processing the first training image and the at least one first interference image using the network to be trained to obtain robust loss includes:

[0014] The first training image is processed using the network to be trained to obtain a first classification result for the first training image;

[0015] Based on the difference between the first classification result and the first category, the first classification loss is obtained;

[0016] The network to be trained is used to process the at least one first interference image to obtain interference loss;

[0017] The robust loss is obtained by weighted summation of the first classification loss and the interference loss.

[0018] In conjunction with any embodiment of this application, obtaining the first training image includes:

[0019] Based on the second annotation data, the pixel region including the detected object is extracted from the second training image to obtain the first training image.

[0020] In any embodiment of this application, the at least one first interference image includes a second interference image and a third interference image that is different from the second interference image;

[0021] The step of processing the at least one first interference image using the network to be trained to obtain interference loss includes:

[0022] The network to be trained is used to perform feature extraction processing on the second interference image and the third interference image respectively to obtain the first feature data of the second interference image and the second feature data of the third interference image;

[0023] The enhancement loss is obtained based on the difference between the first feature data and the second feature data;

[0024] The interference loss is obtained based on the enhancement loss.

[0025] In any embodiment of this application, the at least one first interference image includes a fourth interference image, which is obtained by rotating the first training image by a first angle;

[0026] Before obtaining the interference loss based on the enhancement loss, the method further includes:

[0027] The fourth interference image is processed using the network to be trained to obtain a second angle, which represents the rotation angle of the fourth interference image relative to the first training image.

[0028] The rotational loss is obtained based on the difference between the first angle and the second angle;

[0029] The process of obtaining the interference loss based on the enhancement loss includes:

[0030] The interference loss is obtained based on the enhancement loss and rotation loss.

[0031] In any embodiment of this application, processing the second training image using the network to be trained to obtain the target detection loss includes:

[0032] The second training image is processed using the network to be trained to obtain a second classification result of the detected object and a second position of the detected object in the second training image;

[0033] The second classification loss is obtained based on the difference between the second category and the second classification result;

[0034] The first regression loss is obtained based on the difference between the first position and the second position;

[0035] The target detection loss is obtained based on the second classification loss and the first regression loss.

[0036] In any embodiment of this application, processing the second training image using the network to be trained to obtain a second classification result of the detected object and a second position of the detected object in the second training image includes:

[0037] The second training image is processed using the network to be trained to determine the candidate region of the detection object from the second training image;

[0038] Determine the category of the candidate region to obtain the second classification result;

[0039] The position of the candidate region in the second training image is determined to obtain the second position.

[0040] In conjunction with any embodiment of this application, before obtaining the target detection loss based on the second classification loss and the first regression loss, the method further includes:

[0041] The probability of the candidate region being a foreground region is determined to obtain a second detection result, wherein the foreground region is the pixel region corresponding to the detected object;

[0042] Based on the second detection result, the first position, and the second position, a first interest loss is obtained, and the first interest loss is negatively correlated with the accuracy of the second detection result;

[0043] The method of obtaining the target detection loss based on the second classification loss and the first regression loss includes:

[0044] The target detection loss is obtained based on the second classification loss, the first regression loss, and the first interest loss.

[0045] In conjunction with any embodiment of this application, before obtaining the target detection loss based on the second classification loss, the first regression loss, and the first interest loss, the method further includes:

[0046] The first training image is processed by feature extraction using the network to be trained to obtain the third feature data of the first training image;

[0047] Based on the third feature data, the probability that the first training image is the foreground region is determined, and a third detection result is obtained;

[0048] Based on the difference between the third detection result and the first labeled data, a second loss of interest is obtained;

[0049] The target detection loss is obtained based on the second classification loss, the first regression loss, and the first interest loss, including:

[0050] The target detection loss is obtained based on the second classification loss, the first regression loss, the first interest loss, and the second interest loss.

[0051] In conjunction with any embodiment of this application, before obtaining the target detection loss based on the second classification loss, the first regression loss, the first interest loss, and the second interest loss, the method further includes:

[0052] The candidate region is processed by the network to be trained to obtain the fourth feature data of the candidate region;

[0053] The fourth feature data is subjected to data perturbation to obtain the fifth feature data;

[0054] Based on the fifth feature data, the third classification result of the candidate region is obtained;

[0055] The third classification loss is obtained based on the difference between the third classification result and the second category;

[0056] The target detection loss is obtained based on the second classification loss, the first regression loss, the first interest loss, and the second interest loss, including:

[0057] The target detection loss is obtained based on the second classification loss, the first regression loss, the first interest loss, the second interest loss, and the third classification loss.

[0058] In conjunction with any embodiment of this application, before obtaining the target detection loss based on the second classification loss, the first regression loss, the first interest loss, the second interest loss, and the third classification loss, the method further includes:

[0059] Based on the fifth feature data, the third position of the detected object in the second training image is obtained;

[0060] The second regression loss is obtained based on the difference between the first position and the third position;

[0061] The target detection loss is obtained based on the second classification loss, the first regression loss, the first interest loss, the second interest loss, and the third classification loss, including:

[0062] The target detection loss is obtained based on the second classification loss, the first regression loss, the first interest loss, the second interest loss, the third classification loss, and the second regression loss.

[0063] This application also provides an image processing apparatus, the apparatus comprising:

[0064] An acquisition unit is used to acquire an image to be processed and an object detection network. The object detection network is trained using a first training image, a second training image, and at least one first interference image. The at least one first interference image is obtained by performing image interference processing on the first training image. The first annotation data of the first training image is the first category of the first training image. The second annotation data of the second training image includes the second category of the detected object in the second training image and the first position of the detected object in the second training image.

[0065] The processing unit is used to perform target detection on the image to be processed using the target detection network to obtain a first detection result.

[0066] In conjunction with any embodiment of this application, the acquisition unit is used for:

[0067] Obtain the network to be trained, the first training image, the second training image, and the at least one first interference image;

[0068] The first training image and the at least one first interference image are processed using the network to be trained to obtain a robust loss;

[0069] The target detection loss is obtained by processing the second training image using the network to be trained.

[0070] Based on the target detection loss and the robust loss, the total loss of the network to be trained is obtained;

[0071] Based on the total loss, the parameters of the network to be trained are updated to obtain the target detection network.

[0072] In conjunction with any embodiment of this application, the acquisition unit is used for:

[0073] The first training image is processed using the network to be trained to obtain a first classification result for the first training image;

[0074] Based on the difference between the first classification result and the first category, the first classification loss is obtained;

[0075] The network to be trained is used to process the at least one first interference image to obtain interference loss;

[0076] The robust loss is obtained by weighted summation of the first classification loss and the interference loss.

[0077] In conjunction with any embodiment of this application, the acquisition unit is used for:

[0078] Based on the second annotation data, the pixel region including the detected object is extracted from the second training image to obtain the first training image.

[0079] In any embodiment of this application, the at least one first interference image includes a second interference image and a third interference image that is different from the second interference image;

[0080] The step of processing the at least one first interference image using the network to be trained to obtain interference loss includes:

[0081] The network to be trained is used to perform feature extraction processing on the second interference image and the third interference image respectively to obtain the first feature data of the second interference image and the second feature data of the third interference image;

[0082] The enhancement loss is obtained based on the difference between the first feature data and the second feature data;

[0083] The interference loss is obtained based on the enhancement loss.

[0084] In any embodiment of this application, the at least one first interference image includes a fourth interference image, which is obtained by rotating the first training image by a first angle;

[0085] The acquisition unit is used for:

[0086] The fourth interference image is processed using the network to be trained to obtain a second angle, which represents the rotation angle of the fourth interference image relative to the first training image.

[0087] The rotational loss is obtained based on the difference between the first angle and the second angle;

[0088] The interference loss is obtained based on the enhancement loss and rotation loss.

[0089] In conjunction with any embodiment of this application, the acquisition unit is used for:

[0090] The second training image is processed using the network to be trained to obtain a second classification result of the detected object and a second position of the detected object in the second training image;

[0091] The second classification loss is obtained based on the difference between the second category and the second classification result;

[0092] The first regression loss is obtained based on the difference between the first position and the second position;

[0093] The target detection loss is obtained based on the second classification loss and the first regression loss.

[0094] In conjunction with any embodiment of this application, the acquisition unit is used for:

[0095] The second training image is processed using the network to be trained to determine the candidate region of the detection object from the second training image;

[0096] Determine the category of the candidate region to obtain the second classification result;

[0097] The position of the candidate region in the second training image is determined to obtain the second position.

[0098] In conjunction with any embodiment of this application, the acquisition unit is used for:

[0099] The probability of the candidate region being a foreground region is determined to obtain a second detection result, wherein the foreground region is the pixel region corresponding to the detected object;

[0100] Based on the second detection result, the first position, and the second position, a first interest loss is obtained, and the first interest loss is negatively correlated with the accuracy of the second detection result;

[0101] The target detection loss is obtained based on the second classification loss, the first regression loss, and the first interest loss.

[0102] In conjunction with any embodiment of this application, the acquisition unit is used for:

[0103] The first training image is processed by feature extraction using the network to be trained to obtain the third feature data of the first training image;

[0104] Based on the third feature data, the probability that the first training image is the foreground region is determined, and a third detection result is obtained;

[0105] Based on the difference between the third detection result and the first labeled data, a second loss of interest is obtained;

[0106] The target detection loss is obtained based on the second classification loss, the first regression loss, the first interest loss, and the second interest loss.

[0107] In conjunction with any embodiment of this application, the acquisition unit is used for:

[0108] The candidate region is processed by the network to be trained to obtain the fourth feature data of the candidate region;

[0109] The fourth feature data is subjected to data perturbation to obtain the fifth feature data;

[0110] Based on the fifth feature data, the third classification result of the candidate region is obtained;

[0111] The third classification loss is obtained based on the difference between the third classification result and the second category;

[0112] The target detection loss is obtained based on the second classification loss, the first regression loss, the first interest loss, the second interest loss, and the third classification loss.

[0113] In conjunction with any embodiment of this application, the acquisition unit is used for:

[0114] Based on the fifth feature data, the third position of the detected object in the second training image is obtained;

[0115] The second regression loss is obtained based on the difference between the first position and the third position;

[0116] The target detection loss is obtained based on the second classification loss, the first regression loss, the first interest loss, the second interest loss, the third classification loss, and the second regression loss.

[0117] This application also provides an electronic device, characterized in that it includes: a processor and a memory, the memory being used to store computer program code, the computer program code including computer instructions, wherein, when the processor executes the computer instructions, the electronic device performs a method such as the image processing method described above and any possible implementation thereof.

[0118] This application also provides another electronic device, comprising: a processor, a transmitting device, an input device, an output device, and a memory, the memory being used to store computer program code, the computer program code including computer instructions, wherein, when the processor executes the computer instructions, the electronic device performs a method such as the image processing method described above and any possible implementation thereof.

[0119] This application also provides a computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform a method such as the image processing method described above and any possible implementation thereof.

[0120] This application also provides a computer program product comprising a computer program or instructions that, when the computer program or instructions are executed on a computer, cause the computer to perform the above-described image processing method and any possible implementation thereof.

[0121] In this embodiment, the image processing device uses a second training image to train the target detection network, enabling the network to learn target detection capabilities. At least one first interfering image is obtained by perturbing the first training image. Therefore, while training the deep learning network using the first training image, the target detection network can learn, during training, how to accurately extract feature information from the first training image without perturbing, and how to accurately extract feature information from the first interfering image after perturbing, thereby improving the classification accuracy of the target detection network. Furthermore, since the classification accuracy and target detection accuracy of the target detection network are positively correlated, improving the classification accuracy of the target detection network can improve its target detection accuracy.

[0122] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this application. Attached Figure Description

[0123] To more clearly illustrate the technical solutions in the embodiments of this application or the background art, the accompanying drawings used in the embodiments of this application or the background art will be described below.

[0124] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0125] Figure 1 A schematic flowchart of an image processing method provided in an embodiment of this application;

[0126] Figure 2 This is a schematic diagram of the structure of a network to be trained provided in an embodiment of this application;

[0127] Figure 3 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application;

[0128] Figure 4 This is a schematic diagram of the hardware structure of an image processing device provided in an embodiment of this application. Detailed Implementation

[0129] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0130] The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish different detection objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0131] It should be understood that in this application, "at least one (item)" refers to one or more, "more than" refers to two or more, "at least two (items)" refers to two or three or more, and "and / or" is used to describe the association relationship of the associated detection objects, indicating that there can be three relationships. For example, "A and / or B" can indicate three cases: only A exists, only B exists, and A and B exist simultaneously, where A and B can be singular or plural. The character " / " can indicate that the preceding and following associated detection objects are in an "or" relationship, referring to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can be: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple. The character " / " can also represent the division sign in mathematical operations, for example, a / b = a divided by b; 6 / 3 = 2. "At least one of the following" or similar expressions.

[0132] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0133] Thanks to the rapid development of deep learning technology, the performance of deep learning networks has become increasingly powerful. Therefore, deep learning networks are widely used in image processing, including object detection. However, current deep learning networks used for object detection have poor robustness, leading to low accuracy in object detection results.

[0134] To improve the robustness of deep learning networks, related techniques perform anti-interference training on the deep learning network after completing the object detection training. For example, interference images obtained by image interference processing are used as training data to train the deep learning network.

[0135] This technique, by separating object detection training and anti-interference training, results in poor anti-interference ability of the deep learning network during the training process, thus leading to poor robustness. Therefore, this application provides a technical solution to improve the robustness of deep learning networks.

[0136] The execution subject of this application embodiment is an image processing device, which can be any electronic device capable of executing the technical solutions disclosed in the method embodiments of this application. Optionally, the image processing device can be one of the following: a mobile phone, a computer, a tablet computer, or a wearable smart device.

[0137] It should be understood that the method embodiments of this application can also be implemented by a processor executing computer program code. The embodiments of this application are described below with reference to the accompanying drawings. Please refer to... Figure 1 , Figure 1 This is a flowchart illustrating an image processing method provided in an embodiment of this application.

[0138] 101. Obtain the image to be processed and the object detection network.

[0139] In this embodiment, the image to be processed can contain any content. For example, the image to be processed may include people, motor vehicles, and non-motor vehicles, etc. This application does not limit the content in the image to be processed.

[0140] In one implementation of acquiring an image to be processed, the image processing device receives the image to be processed from input by a user through an input component. Optionally, the input component is one of the following: a keyboard, a mouse, a touchscreen, a touchpad, and an audio input device.

[0141] In another implementation of acquiring the image to be processed, the image processing device receives the image to be processed from the terminal. Optionally, the terminal is one of the following: a mobile phone, a computer, a tablet computer, a server, or a wearable device.

[0142] In another implementation of acquiring the image to be processed, the image processing device is equipped with an image acquisition component, which acquires the image to be processed. Optionally, the image acquisition component is a camera.

[0143] In this embodiment of the application, the object detection network is a deep learning network capable of object detection, for example, a neural network used to achieve object detection.

[0144] By processing images using object detection networks, we can determine whether an image contains a target object and its location within the image. For example, if the target object is a cat, processing image 'a' with an object detection network determines that image 'a' contains a cat and that the cat is located within image 'a'. As another example, if the target object is an apple, processing image 'b' with an object detection network determines that image 'b' does not contain an apple.

[0145] In this embodiment of the application, the target detection network is trained using a first training image, a second training image, and at least one first interference image, wherein the at least one first interference image is obtained by performing image interference processing on the first training image.

[0146] In this embodiment of the application, image interference processing of the first training image includes editing the first training image, that is, at least one first interference image is obtained by editing the first training image.

[0147] For example, at least one first interfering image includes image a. Image a is obtained by adjusting the resolution of the first training image. As another example, at least one first interfering image includes image a and image b. Image a is obtained by rotating the first training image, and image b is obtained by erasing regions from the first training image. Yet another example, at least one first interfering image includes image a and image b. Image a is obtained by rotating the first training image counterclockwise by 90 degrees, and image b is obtained by rotating the first training image counterclockwise by 180 degrees.

[0148] Optionally, image interference processing can be performed using one of the following methods: erasure, cropping, color adjustment, or horizontal flipping. In this embodiment, erasure refers to erasing any pixel region from the image. Cropping refers to cropping a pixel region from the image. Color adjustment refers to adjusting the colors of the image. A mirror effect can be achieved by horizontally flipping the image; that is, horizontally flipping the training image yields a mirror image of the training image.

[0149] In this embodiment, the first annotation data of the first training image is the first category of the first training image. For example, if the first training image includes a cat, then the category of the first training image is cat; if the first training image includes an apple, then the category of the first training image is apple.

[0150] In this embodiment of the application, the second annotation data of the second training image includes the second category of the detected object in the second training image and the first position of the detected object in the second training image, that is, the second annotation data includes the second category and the first position.

[0151] For example, if a dog in the second training image is used as the detection object, then the second category is "dog" and the first position is the position of the dog in the second training image.

[0152] In this embodiment, during the training of the object detection network, a first training image and at least one first interference image are used to train the image classification ability of the object detection network. The at least one first interference image is used to improve the anti-interference ability of the object detection network when performing image classification. Simultaneously, a second training image is also used during the training process to train the object detection ability of the object detection network.

[0153] In one possible implementation, the category of any first interference image is the same as the category of the first training image. During the training of the object detection network, the object detection network is used to process the first training image to obtain a first training classification result for the first training image. The object detection network is then used to process each first interference image to obtain at least one second training classification result for at least one first interference image. The object detection network is then used to process the second training image to obtain the first training category of the detected object in the second training image, and the first training position of the detected object in the second training image.

[0154] A first training loss is obtained based on the difference between the first training classification result and the first category. At least one second training loss is obtained by calculating the difference between each second training classification result and the category of the first interfering image. A third training loss is obtained based on the difference between the first training category and the second category. A fourth training loss is obtained based on the difference between the first position and the second training position.

[0155] Calculate the sum of the first training loss, at least one second training loss, the third training loss, and the fourth training loss to obtain the fifth training loss. Update the parameters of the object detection network based on the fifth training loss until the fifth training loss converges, thus completing the training of the object detection network.

[0156] In another implementation of the training object detection network, the object detection network is used to extract features from the first training image to obtain first training feature data. Based on the first training feature data, a third training classification result for the first training image is obtained. The object detection network is then used to extract features from each of the first interference images to obtain at least one second training feature data for at least one of the first interference images. Finally, the object detection network is used to process the second training image to obtain a second training category for the detected object in the second training image, and a third training position for the detected object in the second training image.

[0157] A sixth training loss is derived based on the difference between the third training classification result and the first category. At least one seventh training loss is derived by calculating the difference between each second training feature data point and the first training feature data point. An eighth training loss is derived based on the difference between the second training categories. A ninth training loss is derived based on the difference between the first position and the third training position.

[0158] Calculate the sum of the sixth training loss, at least one seventh training loss, the eighth training loss, and the ninth training loss to obtain the tenth training loss. Update the parameters of the object detection network based on the tenth training loss until the tenth training loss converges, thus completing the training of the object detection network.

[0159] It should be understood that in the implementation of this application, the steps of acquiring the image to be processed and acquiring the target detection network can be performed simultaneously or separately, and this application does not limit this.

[0160] 102. Using the above-mentioned target detection network, target detection is performed on the above-mentioned image to be processed to obtain the first detection result.

[0161] In this embodiment, the first detection result includes the position of the target object in the image to be processed, wherein the target object is the detected object of the target detection. For example, if the detected object of the target detection is an apple, then the first detection result includes the position of the apple in the image to be processed. As another example, if the detected objects of the target detection include cats and dogs, then the first detection result includes the positions of the cat and the dog in the image to be processed.

[0162] In this embodiment, the image processing device uses a second training image to train the target detection network, enabling the network to learn target detection capabilities. At least one first interfering image is obtained by perturbing the first training image. Therefore, while training the deep learning network using the first training image, the target detection network can learn, during training, how to accurately extract feature information from the first training image without perturbing, and how to accurately extract feature information from the first interfering image after perturbing, thereby improving the classification accuracy of the target detection network. Furthermore, since the classification accuracy and target detection accuracy of the target detection network are positively correlated, improving the classification accuracy of the target detection network can improve its target detection accuracy.

[0163] As an optional implementation, the image processing apparatus acquires the object detection network by performing the following steps:

[0164] 1. Obtain the network to be trained, the first training image, the second training image, and at least one first interference image.

[0165] In this embodiment, the network to be trained is any deep learning network. Optionally, the network to be trained is any neural network. For example, the network to be trained may be composed of at least one network layer stacked together, including convolutional layers, pooling layers, normalization layers, fully connected layers, downsampling layers, upsampling layers, and classifiers. This embodiment does not limit the structure of the network to be trained.

[0166] In one implementation of acquiring a network to be trained, the image processing device receives the network to be trained from input by a user through an input component.

[0167] In another implementation of acquiring the network to be trained, the image processing device receives the network to be trained sent by the terminal.

[0168] In one implementation of acquiring a first training image, the image processing device receives the first training image input by a user through an input component.

[0169] In another implementation of acquiring the first training image, the image processing device receives the first training image sent by the terminal.

[0170] In one implementation of acquiring a second training image, the image processing device receives the second training image input by a user through an input component.

[0171] In another implementation of acquiring the second training image, the image processing device receives the second training image sent by the terminal.

[0172] In one implementation of acquiring at least one first interference image, the image processing device receives at least one first interference image input by a user through an input component.

[0173] In another implementation of acquiring at least one first interference image, the image processing device receives at least one first interference image sent by the terminal.

[0174] It should be understood that in the implementation of this application, the steps of obtaining the network to be trained, obtaining the first training image, obtaining the second training image, and obtaining at least one first interference image can be performed simultaneously or separately, and this application does not limit this.

[0175] 2. The first training image and the at least one first interference image are processed using the network to be trained to obtain a robust loss.

[0176] In this embodiment, robustness loss is used to improve the robustness of the network under training when performing image classification. In one possible implementation, the image processing device obtains the robustness loss by performing steps 3 to 6:

[0177] 3. The first training image is processed using the network to be trained to obtain a first classification result for the first training image. The image processing device processes the first training image using the network to be trained to determine the category of the first training image and obtain a first classification result.

[0178] In one possible implementation, the first classification result includes the category of the first training image. For example, the network to be trained is used to distinguish whether an image belongs to the category of apple, pear, or orange. The image processing device processes the first training image using the network to be trained and determines that the category of the first training image is apple. In this case, the first classification result is apple.

[0179] In another possible implementation, the first classification result includes the category of the first training image and the confidence level of that category. For example, the network to be trained is used to distinguish whether an image is an apple, a pear, or an orange. The image processing device processes the first training image using the network to be trained, determining that the probability of the first training image being an apple is 0.2, the probability of it being a pear is 0.7, and the probability of it being an orange is 0.1. Since the probability of the first training image being a pear is the highest, the first training image is classified as a pear, and the confidence level for the pear in the first training image is 0.7.

[0180] 4. Based on the difference between the first classification result and the first category, the first classification loss is obtained.

[0181] In this embodiment, the greater the difference between the first classification result and the category of the first training image, the greater the first classification loss. In one possible implementation, the image processing device obtains the first classification loss by calculating the cross entropy loss between the first classification result and the category of the first training image.

[0182] In another possible implementation, the image processing device obtains the first classification loss by calculating the negative log-likelihood loss between the first classification result and the category of the first training image.

[0183] In another possible implementation, the image processing device obtains the first classification loss by calculating the exponential loss between the first classification result and the category of the first training image.

[0184] 5. Use the network to be trained to process at least one of the first interference images to obtain interference loss.

[0185] The image processing device determines the method for calculating the first intermediate loss corresponding to the first interfering image based on the image interference processing implementation method used to obtain the first interfering image. The image processing device processes each first interfering image using the network to be trained, obtaining at least one first intermediate loss. The image processing device calculates the sum of the at least one intermediate loss to obtain the interference loss.

[0186] In one possible implementation, the first interfering image is obtained by rotating the first training image by a third angle. The image processing device processes the first interfering image using the network to be trained to obtain a fourth angle, where the fourth angle represents the rotation angle of the first interfering image relative to the first training image. The image processing device calculates the difference between the third angle and the fourth angle to obtain a first intermediate loss of the first interfering image.

[0187] In one possible implementation, the first interfering image is obtained by transforming the scale of the first training image to a third scale. The image processing device processes the first interfering image using the network to be trained to obtain a fourth scale of the first interfering image. The image processing device calculates the difference between the third scale and the fourth scale to obtain a first intermediate loss of the first interfering image.

[0188] 6. The first category loss and the interference loss are weighted and summed to obtain the robust loss.

[0189] 7. The second training image is processed using the network to be trained to obtain the target detection loss.

[0190] In this embodiment, the object detection loss is used to train the object detection capability of the network to be trained. In one possible implementation, the network to be trained processes a second training image to obtain a first intermediate classification result of the detected object in the second training image, and a first intermediate position of the detected object in the second training image. The difference between the first intermediate classification result and the second category is calculated to obtain a second intermediate loss. The difference between the first intermediate position and the first position is calculated to obtain a third intermediate loss. The second intermediate loss and the third intermediate loss are weighted and summed to obtain the object detection loss.

[0191] In another possible implementation, the second training image is processed using the network to be trained to obtain the second intermediate position of the object to be detected in the second training image. The relative entropy loss (KL loss) between the second intermediate position and the first position is calculated to obtain the object detection loss.

[0192] In another possible implementation, the network to be trained processes the second training image to obtain the third intermediate position of the object to be detected in the second training image. The focal loss between the third intermediate position and the first position is calculated to obtain the object detection loss.

[0193] 8. Based on the above target detection loss and the above robust loss, the total loss of the network to be trained is obtained.

[0194] In this embodiment, the total loss of the network to be trained is positively correlated with the target detection loss, and the total loss of the network to be trained is positively correlated with the robustness loss.

[0195] Optionally, the robustness loss is obtained by weighted summation of the first classification loss and the interference loss. The image processing device obtains the total loss of the network to be trained based on the object detection loss, the first classification loss, and the interference loss. In this case, the total loss of the network to be trained is positively correlated with the first classification loss, positively correlated with the object detection loss, and positively correlated with the interference loss. The image processing device can obtain the total loss of the network to be trained by calculating the weighted sum of the first classification loss, the object detection loss, and the interference loss.

[0196] In one possible implementation, assume the total loss of the network to be trained is L. t The first classification loss is L c Target detection loss L e Interference loss is L d L t L c L e L d Satisfy the following formula:

[0197] L t =k1L c +k2L e +k3L d …Formula (1)

[0198] Where k1, k2, and k3 are all real numbers. Optionally, k1 = k2 = k3 = 1.

[0199] Optionally, the image processing device can also reasonably transform formula (1) to determine the implementation method of obtaining the total loss based on the first classification loss, target detection loss and interference loss. That is, the image processing device performs a weighted summation of the first classification loss and interference loss to obtain the first weighted loss, and the image processing device obtains the total loss based on the first weighted loss, wherein the total loss is positively correlated with the first weighted loss.

[0200] In another possible implementation, assume the total loss of the network to be trained is L. t The first classification loss is L c Target detection loss L e Interference loss is L d L t L c L e L d Satisfy the following formula:

[0201]

[0202] Where k1, k2, and k3 are all real numbers. Optionally, k1 = k2 = k3 = 1.

[0203] In another possible implementation, assume the total loss of the network to be trained is L. t The first classification loss is L c Target detection loss L e Interference loss is L d L t L c L e L d Satisfy the following formula:

[0204] L t =(k1L c +k2L e +k3L d ) 2 …Formula (3)

[0205] Where k1, k2, and k3 are all real numbers. Optionally, k1 = k2 = k3 = 1.

[0206] 9. Based on the total loss mentioned above, update the parameters of the network to be trained to obtain the target detection network.

[0207] The parameters of the network to be trained are updated based on the total loss until the total loss converges, thus completing the training of the network to be trained and obtaining the object detection network.

[0208] As an optional implementation, the image processing apparatus acquires the first training image by performing the following steps:

[0209] 10. Based on the second annotation data mentioned above, the pixel region including the detected object is extracted from the second training image to obtain the first training image.

[0210] Since the second annotation data includes the first position of the detected object in the second training image, the image processing device can then determine the pixel region including the detected object based on the first position, and extract the pixel region from the second training image to obtain the first training image.

[0211] Optionally, the first position is the location of the detection box of the detected object in the second training image. Based on the first position in the second annotation, the image processing device extracts the pixel region enclosed by the detection box of the detected object from the second training image to obtain the first training image.

[0212] It should be understood that since the category of the detected object is the second category, when the first training image is obtained by cropping the pixel region containing the detected object from the second training image, the category of the first training image is the second category, that is, the first category and the second category are the same at this time.

[0213] For example, the second training image includes detection object a and detection object b, where detection object a is classified as a cow and detection object b is classified as a sheep. When the image processing device extracts a pixel region including detection object a from the second training image to obtain the first training image, the category of the first training image is cow, meaning both the first and second categories are cows. Similarly, when the image processing device extracts a pixel region including detection object b from the second training image to obtain the first training image, the category of the first training image is sheep, meaning both the first and second categories are sheep.

[0214] In this embodiment of the application, the image processing device can reduce the cost of obtaining the first training image by performing step 10.

[0215] As an optional implementation, the aforementioned at least one first interfering image includes a second interfering image and a third interfering image different from the second interfering image. Both the second and third interfering images are obtained by performing image content interference processing on the first training image, wherein the image content interference processing is used to change the image content of the first training image.

[0216] In one possible implementation, image content interference is handled by erasing. In another possible implementation, image content interference is handled by changing the color of the image.

[0217] Optionally, the second and third interfering images are obtained by performing different image content interference processing on the first training image. For example, the second interfering image is obtained by changing the color of the first training image, and the third interfering image is obtained by erasing a portion of the pixel region in the first training image. Alternatively, the second interfering image is obtained by erasing a first pixel region in the first training image, and the third interfering image is obtained by erasing a second pixel region in the first training image, wherein the position of the first pixel region in the first training image is different from the position of the second pixel region in the first training image.

[0218] The image processing device performs the following steps during step 5:

[0219] 11. Using the network to be trained, feature extraction processing is performed on the second interference image and the third interference image respectively to obtain the first feature data of the second interference image and the second feature data of the third interference image.

[0220] The image processing device uses the network to be trained to perform feature extraction processing on the second interfering image to obtain first feature data. The image processing device uses the network to be trained to perform feature extraction processing on the third interfering image to obtain second feature data.

[0221] 12. Based on the difference between the first feature data and the second feature data, the enhancement loss is obtained.

[0222] In this embodiment, the enhancement loss is used to measure the difference between the first feature data and the second feature data, wherein the difference is positively correlated with the enhancement loss. That is, the smaller the difference between the first feature data and the second feature data, the greater the similarity between the first feature data and the second feature data, and the smaller the enhancement loss.

[0223] In one possible implementation, the image processing device obtains a first classification loss by calculating the Euclidean distance between the first feature data and the second feature data.

[0224] In another possible implementation, the image processing device obtains the first classification loss by calculating the Mahalanobis distance between the first feature data and the second feature data.

[0225] In another possible implementation, the image processing device obtains the first classification loss by calculating the Chebyshev distance between the first feature data and the second feature data.

[0226] 13. Based on the above enhancement loss, the above interference loss is obtained.

[0227] In this embodiment, the interference loss and enhancement loss are positively correlated. In one possible implementation, the image processing device treats the enhancement loss as the interference loss. Assume the enhancement loss is L. a The interference loss is L d Then L a L d Satisfy the following formula:

[0228] L d =L a …Formula (4)

[0229] Optionally, the image processing device can also reasonably transform formula (4) to determine the basis enhancement loss and obtain the method of realizing interference loss.

[0230] In one possible implementation, assume the enhancement loss is L. a The interference loss is L d Then L a L d Satisfy the following formula:

[0231] L d =k4×L a …Formula (5)

[0232] Where k4 is a positive number.

[0233] In another possible implementation, assume the enhancement loss is La The interference loss is L d Then L a L d Satisfy the following formula:

[0234]

[0235] Where s is a real number.

[0236] In steps 11-13, both the second and third interfering images are obtained by processing the first training image with image content perturbation. The categories of the second and third interfering images are the same as those of the first training image; that is, both the second and third interfering images are positive samples. Therefore, when the first feature data of the second interfering image is extracted using the network under training, and the second feature data of the third interfering image is extracted using the network under training, calculating the difference between the first and second feature data is equivalent to calculating the difference between the feature data of two positive samples. Therefore, the enhancement loss obtained by executing steps 11-13 can be used to measure the difference between the feature data of two positive samples that have undergone image content perturbation processing.

[0237] Therefore, by obtaining the interference loss based on the enhancement loss and updating the network to be trained through the interference loss (i.e., obtaining the total loss based on the interference loss, the first classification loss, and the object detection loss, and updating the parameters of the network to be trained based on the total loss), the accuracy of the feature data extracted by the network to be trained from the first training image after the first training image content interference processing can be improved, thereby improving the classification accuracy of the first training image after the first training image content interference processing, and thus improving the object detection accuracy of the network to be trained.

[0238] As an optional implementation, the aforementioned at least one first interference image includes a fourth interference image, which is obtained by rotating the aforementioned first training image. Optionally, the fifth interference image is obtained by rotating the training image counterclockwise by a first angle.

[0239] Before performing step 13, the image processing device also performs the following steps:

[0240] 14. The above-mentioned network to be trained is used to process the fourth interference image to obtain the second angle.

[0241] In this embodiment, the second angle represents the rotation angle of the fifth interference image relative to the training image. That is, by processing the fifth interference image using the network to be trained, it is determined that rotating the training image by the second angle yields the fifth interference image. Optionally, the fifth interference image can be obtained by rotating the training image counterclockwise by the second angle.

[0242] 15. Based on the difference between the first angle and the second angle, the rotational loss is obtained.

[0243] In this embodiment, the greater the difference between the first angle and the second angle, the greater the rotation loss. Given the rotation loss, the image processing device performs the following steps during step 13:

[0244] 16. Based on the above-mentioned enhancement loss and rotational loss, the above-mentioned interference loss is obtained.

[0245] In this embodiment, the interference loss is positively correlated with the rotation loss. In one possible implementation, the image processing device can obtain the interference loss by weighted summing of the enhancement loss and the rotation loss.

[0246] In one possible implementation, assume the enhancement loss is L. a The rotational loss is L r The interference loss is L d Then L a L r L d Satisfy the following formula:

[0247] L d =k5L a +k6L r …Formula (7)

[0248] Where k5 and k6 are both real numbers. Optionally, k5 = k6 = 1.

[0249] Optionally, the image processing device can also reasonably modify formula (7) to determine the method of obtaining the interference loss based on the enhancement loss and rotation loss. That is, the image processing device performs a weighted summation of the enhancement loss and rotation loss to obtain the second weighted loss, and the image processing device obtains the interference loss based on the second weighted loss, wherein the interference loss is positively correlated with the second weighted loss.

[0250] In another possible implementation, assume the enhancement loss is L a The rotational loss is L r The interference loss is L d Then L a L r L d Satisfy the following formula:

[0251]

[0252] Where k5 and k6 are both real numbers. Optionally, k5 = k6 = 1.

[0253] In another possible implementation, assume the enhancement loss is L. aThe rotational loss is L r The interference loss is L d Then L a L r L d Satisfy the following formula:

[0254] L d =(k5L a +k6L r ) 2 …Formula (9)

[0255] Where k5 and k6 are both real numbers. Optionally, k5 = k6 = 1.

[0256] In steps 14 to 16, the image processing device, after determining the second angle of the fourth interfering image using the network to be trained, calculates the rotation loss obtained by the difference between the first angle and the second angle, which can be used to measure the accuracy of the network to be trained in predicting the rotation angle of the image.

[0257] Therefore, by obtaining the interference loss based on the enhancement loss and rotation loss, and updating the network to be trained using the interference loss (i.e., obtaining the total loss based on the interference loss, the first classification loss, and the object detection loss, and updating the parameters of the network to be trained based on the total loss), the classification accuracy of the first training image can be improved when the first training image has undergone image content interference processing, and the classification accuracy of the first training image can also be improved when the first training image has been rotated, thereby improving the object detection accuracy of the network to be trained.

[0258] As an optional implementation, the image processing device performs the following steps during step 7:

[0259] 17. The second training image is processed using the network to be trained to obtain the second classification result of the detected object and the second position of the detected object in the second training image.

[0260] The network to be trained processes the second training image to determine the category of the detected object to obtain the second classification result, and determines the position of the detected object in the second training image to obtain the second position.

[0261] 18. Based on the difference between the second category and the second classification result, the second classification loss is obtained.

[0262] In this embodiment, the greater the difference between the second classification result and the second category, the greater the second classification loss. In one possible implementation, the image processing device obtains the second classification loss by calculating the cross-entropy loss between the second classification result and the category of the training image.

[0263] In another possible implementation, the image processing device obtains the second classification loss by calculating the negative log-likelihood loss between the second classification result and the category of the training image.

[0264] In another possible implementation, the image processing device obtains the second classification loss by calculating the exponential loss between the second classification result and the category of the training image.

[0265] 19. Based on the difference between the first position and the second position, the first regression loss is obtained.

[0266] In this embodiment of the application, the greater the difference between the first position and the second position, the greater the first regression loss. In one possible implementation, the image processing device calculates the intersection-union ratio (IoULoss) loss between the first position and the second position to obtain the first regression loss.

[0267] In another possible implementation, the image processing device calculates the mean absolute error loss (L1 Loss) between the first position and the second position to obtain the first regression loss.

[0268] In another possible implementation, the image processing device calculates the mean squared error loss (L2 Loss) between the first position and the second position to obtain the first regression loss.

[0269] 20. Based on the above-mentioned second classification loss and the above-mentioned first regression loss, the above-mentioned target detection loss is obtained.

[0270] In this embodiment, the target detection loss is positively correlated with the second classification loss and positively correlated with the first regression loss. The image processing device can obtain the target detection loss by calculating the weighted sum of the second classification loss and the first regression loss.

[0271] In one possible implementation, assume the object detection loss is L. e The second classification loss is L f The first regression loss is L g L e L f L g Satisfy the following formula:

[0272] L e =k7L f +k8L g …Formula (10)

[0273] Where k7 and k8 are both real numbers. Optionally, k7 = k8 = 1.

[0274] Optionally, the image processing device can also reasonably transform formula (10) to determine the implementation method of obtaining the target detection loss based on the second classification loss and the first regression loss. That is, the image processing device performs a weighted summation of the second classification loss and the first regression loss to obtain the third weighted loss, and the image processing device obtains the target detection loss based on the third weighted loss, wherein the target detection loss is positively correlated with the third weighted loss.

[0275] In another possible implementation, assume the object detection loss is L. e The second classification loss is L f The first regression loss is L g L e L f L g Satisfy the following formula:

[0276]

[0277] Where k7 and k8 are both real numbers. Optionally, k7 = k8 = 1.

[0278] In another possible implementation, suppose the object detection loss is L e The second classification loss is L f The first regression loss is L g L e L f L g Satisfy the following formula:

[0279] Le=(k7L f +k8L g ) 2 …Formula (12)

[0280] Where k7 and k8 are both real numbers. Optionally, k7 = k8 = 1.

[0281] As an optional implementation, the image processing apparatus performs the following steps during step 17:

[0282] 21. The second training image is processed using the network to be trained to determine the candidate region of the detection object from the second training image.

[0283] In this embodiment, the candidate region is the pixel region containing the detection object determined by the network to be trained. That is, the network to be trained processes the second training image to determine the pixel region containing the detection object from the second training image, thus obtaining the candidate region of the detection object.

[0284] Optionally, the network to be trained processes the second training image to determine candidate boxes for detecting objects from the second training image, and takes the pixel region enclosed by the candidate box as the candidate region, wherein the candidate box is the box that includes the object to be detected, as determined by the network to be trained.

[0285] 22. Determine the category of the above candidate regions to obtain the second classification result.

[0286] The network to be trained obtains a second classification result by determining the category of the detected object in the candidate region.

[0287] 23. Determine the position of the above candidate region in the above second training image to obtain the above second position.

[0288] In this embodiment, when the network to be trained determines candidate regions of the detected object from the second training image, it obtains a second classification result by determining the category of the candidate regions. This reduces the amount of data processing required to determine the second classification result and improves its accuracy. Furthermore, by determining the position of the candidate regions in the second training image, the position of the detected object in the second training image (i.e., the second position) can be obtained.

[0289] As an optional implementation, the image processing apparatus further performs the following steps before performing step 20:

[0290] 24. Determine the probability that the above candidate regions are foreground regions to obtain the second detection result.

[0291] In this embodiment, the foreground region is the pixel region corresponding to the detected object, that is, the pixel region covered by the detected object. For example, if the detected object is a cow, the foreground region is the pixel region covered by the cow; if the detected object is a person, the foreground region is the pixel region covered by the person.

[0292] In one possible implementation, the image processing device determines the probability that the candidate region includes the foreground region, thus obtaining a second detection result.

[0293] 25. Based on the above second detection result, the above first position and the above second position, the first loss of interest is obtained.

[0294] In this embodiment, the first loss of interest is negatively correlated with the second detection result; that is, the lower the accuracy of the second detection result, the greater the first loss of interest.

[0295] Since the first position is the location of the detected object in the second training image, the pixel region determined from the second training image based on the first position (hereinafter referred to as the pixel region corresponding to the first position) is the foreground region, while all pixel regions in the second training image other than the pixel region corresponding to the first position are non-foreground regions.

[0296] Therefore, the image processing device can determine whether the candidate region is a foreground region or a background region based on the first position and the second position. That is, the image processing device determines the third annotation data of the candidate region based on the first position and the second position, wherein the third annotation data includes whether the candidate region is a foreground region or a non-foreground region.

[0297] In one possible implementation, the image processing device determines the target intersection-over-union ratio (IoU) between the candidate region and the pixel region corresponding to the first position based on a first position and a second position. If the target IoU is greater than or equal to an IoU threshold, the candidate region is determined to be a foreground region; if the target IoU is less than an IoU threshold, the candidate region is determined to be a non-foreground region.

[0298] Based on the difference between the second detection result and the third annotation data, the image processing device can determine the accuracy of the second detection result and thus obtain the first loss of interest.

[0299] In one possible implementation, if the third labeled data includes a candidate region that is a foreground region, the probability that the third labeled data includes a candidate region that is a foreground region is 1. The image processing device calculates the difference between the third labeled data and the second detection result to obtain a first interest loss, wherein the larger the difference between the third labeled data and the second detection result, the larger the first interest loss.

[0300] In another possible implementation, if the third annotation data includes a candidate region that is a foreground region, the probability that the third annotation data includes a candidate region that is a foreground region is 1. When the third annotation data is the same as the second detection result, the image processing device determines the first interest loss as a first value; when the third annotation data is different from the second detection result, the image processing device determines the first interest loss as a second value, wherein the second value is greater than the first value.

[0301] For example, the first value is 0, and the second value is 3. If the second detection result is 0.8, and the probability that the third labeled data includes a candidate region that is a foreground region is 1, then the second detection result is different from the third labeled data, and the first interest loss is 3. If the second detection result is 1, and the probability that the third labeled data includes a candidate region that is a foreground region is 1, then the second detection result is the same as the third labeled data, and the first interest loss is 0.

[0302] Having obtained the first loss of interest, the image processing device performs the following steps during step 20:

[0303] 26. Based on the above-mentioned second classification loss, the above-mentioned first regression loss, and the above-mentioned first interest loss, the above-mentioned target detection loss is obtained.

[0304] In this embodiment, the target detection loss is positively correlated with the second classification loss, positively correlated with the first regression loss, and positively correlated with the first interest loss. The image processing device obtains the target detection loss by calculating the weighted sum of the second classification loss, the first regression loss, and the first interest loss.

[0305] In one possible implementation, assume the object detection loss is L. e The second classification loss is L f The first regression loss is L g The first interest loss is L x L e L f L g L r Satisfy the following formula:

[0306] L e =k9L f +k 10 L g +k 11 L x …Formula (13)

[0307] Among them, k9, k 10 k 11 All are real numbers. Optional, k9 = k 10 =k 11 =1.

[0308] Optionally, the image processing device can also reasonably transform formula (13) to determine the implementation method of obtaining the target detection loss based on the second classification loss, the first regression loss, and the first interest loss. That is, the image processing device performs a weighted summation of the second classification loss, the first regression loss, and the first interest loss to obtain the fourth weighted loss, and the image processing device obtains the target detection loss based on the fourth weighted loss, wherein the target detection loss is positively correlated with the fourth weighted loss.

[0309] In another possible implementation, assume the object detection loss is L. e The second classification loss is L f The first regression loss is L g The first interest loss is L x L e L f L g L r Satisfy the following formula:

[0310]

[0311] Among them, k9, k 10 k11 All are real numbers. Optional, k9 = k 10 =k 11 =1.

[0312] In another possible implementation, suppose the object detection loss is L e The second classification loss is L f The first regression loss is L g The first interest loss is L x L e L f L g L r Satisfy the following formula:

[0313] L e =(k9L f +k 10 L g +k 11 L x ) 2 …Formula (15)

[0314] Among them, k9, k 10 k 11 All are real numbers. Optional, k9 = k 10 =k 11 =1.

[0315] In steps 24 to 26, when the image processing device obtains a second detection result by using the probability that the candidate region is a foreground region determined by the network to be trained, the first interest loss obtained based on the second detection result, the first position, and the second position can be used to measure the accuracy of the network to be trained in determining the foreground region from the second training image.

[0316] Therefore, by obtaining the object detection loss based on the second classification loss, the first regression loss, and the first interest loss, and updating the network to be trained using the object detection loss (i.e., obtaining the total loss based on the interference loss, the first classification loss, and the object detection loss, and updating the parameters of the network to be trained based on the total loss), the classification accuracy of the first training image can be improved when the first training image has undergone image content interference processing, and the classification accuracy of the first training image when the first training image has been rotated. Furthermore, the accuracy of the network to be trained in identifying the foreground region from the second training image can also be improved. Since the detected object is within the foreground region, improving the accuracy of identifying the foreground region from the second training image further improves the object detection accuracy of the network to be trained.

[0317] As an optional implementation, the image processing apparatus further performs the following steps before performing step 26:

[0318] 27. Using the network to be trained, the first training image is processed to extract features, and the third feature data of the first training image is obtained.

[0319] 28. Based on the third feature data mentioned above, determine the probability that the first training image is a foreground region, and obtain the third detection result.

[0320] In one possible implementation, the image processing device determines the probability that the first training image includes a foreground region, and obtains a second detection result.

[0321] 29. Based on the difference between the third detection result and the first labeled data, the second loss of interest is obtained.

[0322] In this embodiment, the second loss of interest is negatively correlated with the third detection result; that is, the lower the accuracy of the third detection result, the greater the second loss of interest.

[0323] Since the first annotation data includes a first category of the first training image, the image processing device can determine whether the first training image includes a detection object, i.e., whether the first training image includes a foreground region, based on the first category. For example, the detection object is a cow. If the first category is cow, then the image processing device determines that the first training image includes a cow, i.e., the first training image includes a foreground region.

[0324] Based on the difference between the third detection result and the first annotation data, the image processing device can determine the accuracy of the third detection result and then obtain the second loss of interest.

[0325] In one possible implementation, if the first labeled data includes a candidate region that is a foreground region, the probability that the first labeled data includes a candidate region that is a foreground region is 1. The image processing device calculates the difference between the first labeled data and the third detection result to obtain a second interest loss, wherein the larger the difference between the first labeled data and the third detection result, the larger the second interest loss.

[0326] In another possible implementation, if the first labeled data includes a candidate region that is a foreground region, the probability that the first labeled data includes a candidate region that is a foreground region is 1. If the first labeled data is the same as the third detection result, the image processing device determines the second interest loss as a third value; if the first labeled data is different from the third detection result, the image processing device determines the second interest loss as a fourth value, wherein the third value is greater than the fourth value.

[0327] For example, the third value is 0, and the fourth value is 5. If the third detection result is 0.9, the probability that the first labeled data includes a candidate region that is a foreground region is 1. In this case, the third detection result is different from the first labeled data, and the second interest loss is 5. If the third detection result is 1, the probability that the first labeled data includes a candidate region that is a foreground region is 1. In this case, the third detection result is the same as the first labeled data, and the second interest loss is 0.

[0328] Having obtained the second loss of interest, the image processing device performs the following steps during step 26:

[0329] 30. Based on the above-mentioned second classification loss, first regression loss, first interest loss, and second interest loss, the above-mentioned target detection loss is obtained.

[0330] In this embodiment, the target detection loss is positively correlated with the second classification loss, positively correlated with the first regression loss, positively correlated with the first interest loss, and positively correlated with the second interest loss. The image processing device obtains the target detection loss by calculating the weighted sum of the second classification loss, the first regression loss, the first interest loss, and the second interest loss.

[0331] In one possible implementation, assume the object detection loss is L. e The second classification loss is L f The first regression loss is L g The first interest loss is L x The second interest loss is L y L e L f L g L r L y Satisfy the following formula:

[0332] L e =k 12 L f +k 13 L g +k 14 L x +k 15 L y …Formula (16)

[0333] Where, k 12 k 13 k 14 k 15 All are real numbers. Optional, k 12 =k 13 =k 14 =k 15=1.

[0334] Optionally, the image processing device can also reasonably transform formula (16) to determine the implementation method of obtaining the target detection loss based on the second classification loss, the first regression loss, the first interest loss, and the second interest loss. That is, the image processing device performs a weighted summation of the second classification loss, the first regression loss, the first interest loss, and the second interest loss to obtain the fifth weighted loss, and the image processing device obtains the target detection loss based on the fifth weighted loss, wherein the target detection loss is positively correlated with the fifth weighted loss.

[0335] In another possible implementation, assume the object detection loss is L. e The second classification loss is L f The first regression loss is L g The first interest loss is L x The second interest loss is L y L e L f L g L r L y Satisfy the following formula:

[0336]

[0337] Where, k 12 k 13 k 14 k 15 All are real numbers. Optional, k 12 =k 13 =k 14 =k 15 =1.

[0338] In another possible implementation, suppose the object detection loss is L e The second classification loss is L f The first regression loss is L g The first interest loss is L x The second interest loss is L y L e L f L g L r L y Satisfy the following formula:

[0339] L e =(k 12 L f +k 13 L g +k 14 L x +k 15Ly) 2 …Formula (18)

[0340] Where, k 12 k 13 k 14 k 15 All are real numbers. Optional, k 12 =k 13 =k 14 =k 15 =1.

[0341] In steps 27 to 30, when the image processing device obtains a third detection result by using the probability that the candidate region is a foreground region determined by the network to be trained, the second loss of interest obtained based on the third detection result and the first labeled data can be used to measure the accuracy of the network to be trained in determining whether the first training image is a foreground region.

[0342] Therefore, by obtaining the target detection loss based on the second classification loss, the first regression loss, the first interest loss, and the second interest loss, and updating the network to be trained using the target detection loss (i.e., obtaining the total loss based on the interference loss, the first classification loss, and the target detection loss, and updating the parameters of the network to be trained based on the total loss), the accuracy of the network to be trained in determining whether the first training image is a foreground region can be improved, in addition to improving the classification accuracy of the first training image after image content interference processing, improving the classification accuracy of the first training image after rotation, and improving the accuracy of the network to be trained in determining the foreground region from the second training image.

[0343] That is, by using the first interest loss and the second interest loss, the accuracy of the network under training in identifying foreground regions from images can be improved. Since the detected object is within the foreground region, improving the accuracy of identifying the foreground region from the image can further improve the target detection accuracy of the network under training.

[0344] As an optional implementation, the image processing apparatus further performs the following steps before performing step 30:

[0345] 31. Use the above-mentioned network to be trained to perform feature extraction processing on the above-mentioned candidate regions to obtain the fourth feature data of the above-mentioned candidate regions.

[0346] In this embodiment, the fourth feature data carries semantic information of the candidate region. Optionally, the process by which the network to be trained processes the candidate region to obtain the second classification result includes: performing feature extraction processing on the candidate region to obtain the fourth feature data; and obtaining the second classification result based on the fourth feature data.

[0347] 32. Data perturbation is applied to the above fourth feature data to obtain the fifth feature data.

[0348] In this embodiment, data interference is used to interfere with the data. In one possible implementation, data interference of the fourth feature data involves adding noise to the fourth feature data.

[0349] In another possible implementation, data perturbation of the fourth feature data involves swapping any two elements in the fourth feature data. For example, the fourth feature data is vector a, where the first element of vector a is 3, the second element is 5, and the third element is 10. By perturbing the fourth feature data and swapping the first and third elements of vector a, we obtain vector b. Now, the first element of vector b is 10, the second element is 5, and the third element is 3.

[0350] In another possible implementation, perturbation of the fourth feature data involves concatenating the fourth feature data with the perturbation data. For example, the perturbation data is vector a, and the fourth feature data is vector b. Perturbation of the fourth feature data is simply concatenating vector a with vector b.

[0351] 33. Based on the fifth feature data mentioned above, the third classification result of the above candidate regions is obtained.

[0352] The network to be trained determines the category of the candidate region based on the fifth feature data, and obtains the third classification result.

[0353] 34. Based on the difference between the third classification result and the second category, the third classification loss is obtained.

[0354] In this embodiment, the greater the difference between the third classification result and the category of the second training image (i.e., the second category), the greater the second classification loss. In one possible implementation, the image processing device obtains the second classification loss by calculating the cross-entropy loss between the third classification result and the category of the second training image.

[0355] In another possible implementation, the image processing device obtains the second classification loss by calculating the negative log-likelihood loss between the third classification result and the category of the second training image.

[0356] In another possible implementation, the image processing device obtains the second classification loss by calculating the exponential loss between the third classification result and the category of the second training image.

[0357] Given the third classification loss, the image processing device performs the following steps during step 30:

[0358] 35. Based on the above-mentioned second classification loss, the above-mentioned first regression loss, the above-mentioned first interest loss, the above-mentioned second interest loss, and the above-mentioned third classification loss, the above-mentioned target detection loss is obtained.

[0359] In this embodiment, the target detection loss is positively correlated with the second classification loss, positively correlated with the first regression loss, positively correlated with the first interest loss, positively correlated with the second interest loss, and positively correlated with the third classification loss. The image processing device obtains the target detection loss by calculating the weighted sum of the second classification loss, the first regression loss, the first interest loss, the second interest loss, and the third classification loss.

[0360] In one possible implementation, assume the object detection loss is L. e The second classification loss is L f The first regression loss is L g The first interest loss is L x The second interest loss is L y The third category loss is L z L e L f L g L r L y L z Satisfy the following formula:

[0361] L e =k 16 L f +k 17 L g +k 18 L x +k 19 L y +k 20 L y …Formula (19)

[0362] Where, k 16 k 17 k 18 k 19 k 20 All are real numbers. Optional, k 16 =k 17 =k 18 =k 19 =k 20 =1.

[0363] Optionally, the image processing device can also reasonably transform formula (19) to determine the implementation method of obtaining the target detection loss based on the second classification loss, the first regression loss, the first interest loss, the second interest loss, and the third classification loss. That is, the image processing device performs a weighted summation of the second classification loss, the first regression loss, the first interest loss, the second interest loss, and the third classification loss to obtain the sixth weighted loss, and the image processing device obtains the target detection loss based on the sixth weighted loss, wherein the target detection loss is positively correlated with the sixth weighted loss.

[0364] In another possible implementation, assume the object detection loss is L. e The second classification loss is L f The first regression loss is L g The first interest loss is L x The second interest loss is L y The third category loss is L z L e L f L g L r L y L z Satisfy the following formula:

[0365]

[0366] Where, k 16 k 17 k 18 k 19 k 20 All are real numbers. Optional, k 16 =k 17 =k 18 =k 19 =k 20 =1.

[0367] In another possible implementation, suppose the object detection loss is L e The second classification loss is L f The first regression loss is L g The first interest loss is L x The second interest loss is L y The third category loss is L z L e L f L g L r L y L z Satisfy the following formula:

[0368] L e =(k16 L f +k 17 L g +k 18 L x +k 19 L y +k 20 L y ) 2 …Formula (21)

[0369] Where, k 16 k 17 k 18 k 19 k 20 All are real numbers. Optional, k 16 =k 17 =k 18 =k 19 =k 20 =1.

[0370] In steps 31 to 35, the image processing device extracts the fourth feature data of the candidate region using the network to be trained, and obtains the fifth feature data by perturbing the fourth feature data. Based on the fifth feature data, it determines the third classification result, and obtains the third classification loss based on the difference between the third classification result and the category of the second training image. The third classification loss can then be used to measure the accuracy of the category of the second training image determined by the network to be trained after perturbing the feature data extracted from the second training image.

[0371] Therefore, by obtaining the target detection loss based on the second classification loss, the first regression loss, the first interest loss, the second interest loss, and the third classification loss, and updating the network to be trained using the target detection loss (i.e., obtaining the total loss based on the interference loss, the first classification loss, and the target detection loss, and updating the parameters of the network to be trained based on the total loss), the accuracy of classifying the first training image can be improved when the first training image has undergone image content interference processing, as well as when the first training image has been rotated, and the accuracy of the network to be trained in determining the foreground region from the image. Furthermore, the accuracy of classifying the second training image can also be improved when the feature data extracted from the second training image has undergone data interference, thereby improving the target detection accuracy of the network to be trained.

[0372] As an optional implementation, the image processing apparatus further performs the following steps before performing step 35:

[0373] 36. Based on the fifth feature data mentioned above, the third position of the detected object in the second training image is obtained.

[0374] 37. Based on the difference between the first position and the third position mentioned above, the second regression loss is obtained.

[0375] In this embodiment of the application, the greater the difference between the first position and the third position, the greater the second regression loss. In one possible implementation, the image processing device calculates the intersection-union ratio (IoULoss) loss between the first position and the third position to obtain the second regression loss.

[0376] In another possible implementation, the image processing device calculates the mean absolute error loss (L1 Loss) between the first position and the third position to obtain the second regression loss.

[0377] In another possible implementation, the image processing device calculates the mean squared error loss (L2 Loss) between the first position and the third position to obtain the second regression loss.

[0378] Having obtained the second regression loss, the image processing device performs the following steps during step 33:

[0379] 38. Based on the above-mentioned second classification loss, first regression loss, first interest loss, second interest loss, third classification loss and second regression loss, the above-mentioned target detection loss is obtained.

[0380] In this embodiment, the target detection loss is positively correlated with the second classification loss, positively correlated with the first regression loss, positively correlated with the first interest loss, positively correlated with the second interest loss, positively correlated with the third classification loss, and positively correlated with the second regression loss. The image processing device obtains the target detection loss by calculating the weighted sum of the second classification loss, the first regression loss, the first interest loss, the second interest loss, the third classification loss, and the second regression loss.

[0381] In one possible implementation, assume the object detection loss is L. e The second classification loss is L f The first regression loss is L g The first interest loss is L x The second interest loss is L y The third category loss is L z The second regression loss is L n L e L f L g L r L y L z Ln Satisfy the following formula:

[0382] L e =k 21 L f +k 22 L g +k 23 L x +k 24 L y +k 25 L y +k 26 L n …Formula (22)

[0383] Where, k 21 k 22 k 23 k 24 k 25 k 26 All are real numbers. Optional, k 21 =k 22 =k 23 =k 24 =k 25 =k 26 =1.

[0384] Optionally, the image processing device can also reasonably transform formula (22) to determine the implementation method of obtaining the target detection loss based on the second classification loss, the first regression loss, the first interest loss, the second interest loss, the third classification loss, and the second regression loss. That is, the image processing device performs a weighted summation of the second classification loss, the first regression loss, the first interest loss, the second interest loss, the third classification loss, and the second regression loss to obtain the seventh weighted loss. The image processing device obtains the target detection loss based on the seventh weighted loss, wherein the target detection loss is positively correlated with the seventh weighted loss.

[0385] In another possible implementation, assume the object detection loss is L. e The second classification loss is L f The first regression loss is L g The first interest loss is L x The second interest loss is L y The third category loss is L z The second regression loss is L n L e L f L g L r L y L z L n Satisfy the following formula:

[0386]

[0387] Where, k 21 k 22 k 23 k 24 k 25 k 26 All are real numbers. Optional, k 21 =k 22 =k 23 =k 24 =k 25 =k 26 =1.

[0388] In another possible implementation, suppose the object detection loss is L e The second classification loss is L f The first regression loss is L g The first interest loss is L x The second interest loss is L y The third category loss is L z The second regression loss is L n L e L f L g L r L y L z L n Satisfy the following formula:

[0389] L e =(k 21 L f +k 22 L g +k 23 L x +k 24 L y +k 25 L y +k 26 L n ) 2 …Formula (24)

[0390] Where, k 21 k 22 k 23 k 24 k 25 k 26 All are real numbers. Optional, k 21 =k 22 =k 23 =k 24 =k 25 =k 26=1.

[0391] In steps 36 to 38, the image processing device extracts the fourth feature data of the candidate region using the network to be trained, and obtains the fifth feature data by perturbing the fourth feature data. Based on the fifth feature data, the third position is determined, and the second regression loss is obtained based on the difference between the first position and the third position. The second regression loss can then be used to measure the accuracy of the position of the detected object in the second training image determined by the network to be trained after perturbing the feature data extracted from the second training image.

[0392] Therefore, by obtaining the target detection loss based on the second classification loss, the first regression loss, the first interest loss, the second interest loss, the third classification loss, and the second regression loss, and updating the network to be trained using the target detection loss (i.e., obtaining the total loss based on the interference loss, the first classification loss, and the target detection loss, and updating the parameters of the network to be trained based on the total loss), the accuracy of classifying the first training image can be improved when the first training image has undergone image content interference processing, as well as the accuracy of classifying the first training image when the first training image has been rotated, and the accuracy of the network to be trained in determining the foreground region from the image, and the accuracy of classifying the second training image when the feature data extracted from the second training image has undergone data interference. Furthermore, the accuracy of determining the position of the detected object in the second training image when the feature data extracted from the second training image has undergone data interference can be improved, thereby improving the target detection accuracy of the network to be trained.

[0393] As an optional implementation method, the structure of the network to be trained is as follows: Figure 2 As shown. Figure 2 As shown, the network to be trained includes an image interference processing module (i.e., Figure 2 Data augmentation), backbone network (i.e.) Figure 2 Backbone in the middle), feature pyramid module (i.e. Figure 2 In the FPN), the image-level self-supervised module (i.e. Figure 2 Image-level self-supervised (in the context of image-level self-supervised modules) includes: an image content perturbation fully connected layer (i.e., a fully connected layer). Figure 2 InvHead), rotated fully connected layers (i.e. Figure 2 Rot Head in the regression prediction module (i.e. Figure 2 In the RPN), the fully connected layer for object detection (i.e. Figure 2 (Heads in the text).

[0394] The image interference processing module is used to perform image interference processing on the first training image. Specifically, the first training image is obtained by cropping a pixel region containing the detection object from the second training image (i.e.,...). Figure 2 The image is obtained using the Crop algorithm. The backbone network is used for feature extraction from the image, and it can be any network structure capable of feature extraction. The feature pyramid module is used for multi-scale fusion of feature data to enrich the information carried by the feature data. The network structure of the feature pyramid module is a feature pyramid network (FPN). The image content interference fully connected layer, the rotation fully connected layer, and the object detection fully connected layer are all fully connected layers, but any two of them are different.

[0395] For example, the structure of an image content-interference fully connected layer differs from that of a rotation fully connected layer. Also, the parameters of a rotation fully connected layer differ from those of an object detection fully connected layer.

[0396] The input to the image interference processing module is the second training image. The input to the backbone network includes the data output from the image interference processing. Specifically, the image interference processing module performs image content interference processing on the second training image to obtain a second interference image and a third interference image. The backbone network performs feature extraction processing on the second interference image and the third interference image respectively to obtain the first feature data of the second interference image and the second feature data of the third interference image. The image content interference fully connected layer performs a linear transformation on the first feature data and the second feature data to obtain the transformed first feature data and the transformed second feature data. By calculating the difference between the transformed first feature data and the transformed second feature data, the difference between the first feature data and the second feature data is determined, and the enhancement loss (i.e., Figure 2 Invloss in (the context of the loss).

[0397] The image interference processing module obtains a fourth interference image by rotating the first training image by a first angle. The backbone network performs feature extraction processing on the fourth interference image to obtain the sixth feature data of the fourth interference image. The rotation fully connected layer processes the sixth feature data to obtain the second angle. The difference between the first angle and the second angle is calculated to obtain the rotation loss (i.e., Figure 2 Rot loss in (the process).

[0398] The backbone network also includes a first training image as input, meaning the first training image is directly input to the backbone network without being processed by the image interference processing module. The backbone network performs feature extraction on the first training image to obtain the seventh feature data. The seventh feature data is then processed by the feature pyramid module to obtain the third feature data. Based on the third feature data, the probability that the first training image is a foreground region is determined, resulting in the third detection result. Based on the difference between the third detection result and the first labeled data, the second interest loss (i.e., ...) is obtained. Figure 2 The ROI loss in the model is used. Furthermore, the network to be trained also obtains a first classification result for the first training image based on the third feature data. Based on the difference between the first classification result and the first category of the first training image, a first classification loss (i.e., ROI loss) is obtained. Figure 2 Cls loss in (the context of the loss).

[0399] The backbone network also includes a second training image as input. The backbone network performs feature extraction on the second training image to obtain the fourth feature data. The eighth feature data is then processed by the feature pyramid module to determine candidate regions from the second training image. Optionally, the feature pyramid module processing the eighth feature data is the same as the feature pyramid module processing the seventh feature data.

[0400] The network to be trained determines the probability that a candidate region is a foreground region, thus obtaining a second detection result. Based on the second detection result, the first position, and the second position, a first interest loss (i.e., ...) is obtained. Figure 2 (ROI loss in the context). It should be understood that... Figure 2 The ROI loss in the calculation includes the first interest loss and the second interest loss.

[0401] The regression prediction module also performs feature extraction on the candidate regions to obtain the fourth feature data of the candidate regions. This fourth feature data is then processed by the fully connected layer for object detection to obtain the second classification loss (i.e., Figure 2 Cls loss) and first regression loss (i.e. Figure 2 (Reg loss in the context). It should be understood that... Figure 2 The Cls loss in the model includes the first classification loss and the second classification loss.

[0402] The network to be trained also performs data perturbation on the fourth feature data (i.e. Figure 2 The fifth feature data is obtained by processing the ProposalsDisturbance in the target detection fully connected layer. This fifth feature data is then processed to obtain the third classification loss (i.e., the ProposalsDisturbance). Figure 2 Clsconsisitency loss) and second regression loss (i.e. Figure 2 (Regconsistency loss in the middle).

[0403] exist Figure 2 In the process, the backbone network that performs feature extraction on the fourth interference image and the backbone network that performs feature extraction on the second training image share parameters (i.e., Figure 2 (As shown in the Shared). The data perturbation applied to the fourth feature data, the third classification loss, and the second regression loss all belong to the self-supervised module at the regression level (i.e., Figure 2 Proposal-level self-supervised).

[0404] Given the enhancement loss, rotation loss, first classification loss, second classification loss, third classification loss, first interest loss, second interest loss, first regression loss, and second regression loss, the image processing device calculates the total loss of the network to be trained based on these losses. The device then updates the parameters of the network to be trained based on the total loss until it converges, thus completing the training of the network. After training, the backbone network, feature pyramid module, regression prediction module, and fully connected object detection layer in the network to be trained are used as the object detection network.

[0405] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0406] The methods of the embodiments of this application have been described in detail above, and the apparatus of the embodiments of this application is provided below.

[0407] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application. The image processing device 1 includes: an acquisition unit 11 and a processing unit 12, specifically:

[0408] The acquisition unit 11 is used to acquire the image to be processed and the target detection network. The target detection network is trained using a first training image, a second training image and at least one first interference image. The at least one first interference image is obtained by performing image interference processing on the first training image. The first annotation data of the first training image is the first category of the first training image. The second annotation data of the second training image includes the second category of the detected object in the second training image and the first position of the detected object in the second training image.

[0409] The processing unit 12 is used to perform target detection on the image to be processed using the target detection network to obtain a first detection result.

[0410] In any embodiment of this application, the acquisition unit 11 is used for:

[0411] Obtain the network to be trained, the first training image, the second training image, and the at least one first interference image;

[0412] The first training image and the at least one first interference image are processed using the network to be trained to obtain a robust loss;

[0413] The target detection loss is obtained by processing the second training image using the network to be trained.

[0414] Based on the target detection loss and the robust loss, the total loss of the network to be trained is obtained;

[0415] Based on the total loss, the parameters of the network to be trained are updated to obtain the target detection network.

[0416] In any embodiment of this application, the acquisition unit 11 is used for:

[0417] The first training image is processed using the network to be trained to obtain a first classification result for the first training image;

[0418] Based on the difference between the first classification result and the first category, the first classification loss is obtained;

[0419] The network to be trained is used to process the at least one first interference image to obtain interference loss;

[0420] The robust loss is obtained by weighted summation of the first classification loss and the interference loss.

[0421] In any embodiment of this application, the acquisition unit 11 is used for:

[0422] Based on the second annotation data, the pixel region including the detected object is extracted from the second training image to obtain the first training image.

[0423] In any embodiment of this application, the at least one first interference image includes a second interference image and a third interference image that is different from the second interference image;

[0424] The step of processing the at least one first interference image using the network to be trained to obtain interference loss includes:

[0425] The network to be trained is used to perform feature extraction processing on the second interference image and the third interference image respectively to obtain the first feature data of the second interference image and the second feature data of the third interference image;

[0426] The enhancement loss is obtained based on the difference between the first feature data and the second feature data;

[0427] The interference loss is obtained based on the enhancement loss.

[0428] In any embodiment of this application, the at least one first interference image includes a fourth interference image, which is obtained by rotating the first training image by a first angle;

[0429] The acquisition unit 11 is used for:

[0430] The fourth interference image is processed using the network to be trained to obtain a second angle, which represents the rotation angle of the fourth interference image relative to the first training image.

[0431] The rotational loss is obtained based on the difference between the first angle and the second angle;

[0432] The interference loss is obtained based on the enhancement loss and rotation loss.

[0433] In any embodiment of this application, the acquisition unit 11 is used for:

[0434] The second training image is processed using the network to be trained to obtain a second classification result of the detected object and a second position of the detected object in the second training image;

[0435] The second classification loss is obtained based on the difference between the second category and the second classification result;

[0436] The first regression loss is obtained based on the difference between the first position and the second position;

[0437] The target detection loss is obtained based on the second classification loss and the first regression loss.

[0438] In any embodiment of this application, the acquisition unit 11 is used for:

[0439] The second training image is processed using the network to be trained to determine the candidate region of the detection object from the second training image;

[0440] Determine the category of the candidate region to obtain the second classification result;

[0441] The position of the candidate region in the second training image is determined to obtain the second position.

[0442] In any embodiment of this application, the acquisition unit 11 is used for:

[0443] The probability of the candidate region being a foreground region is determined to obtain a second detection result, wherein the foreground region is the pixel region corresponding to the detected object;

[0444] Based on the second detection result, the first position, and the second position, a first interest loss is obtained, and the first interest loss is negatively correlated with the accuracy of the second detection result;

[0445] The target detection loss is obtained based on the second classification loss, the first regression loss, and the first interest loss.

[0446] In any embodiment of this application, the acquisition unit 11 is used for:

[0447] The first training image is processed by feature extraction using the network to be trained to obtain the third feature data of the first training image;

[0448] Based on the third feature data, the probability that the first training image is the foreground region is determined, and a third detection result is obtained;

[0449] Based on the difference between the third detection result and the first labeled data, a second loss of interest is obtained;

[0450] The target detection loss is obtained based on the second classification loss, the first regression loss, the first interest loss, and the second interest loss.

[0451] In any embodiment of this application, the acquisition unit 11 is used for:

[0452] The candidate region is processed by the network to be trained to obtain the fourth feature data of the candidate region;

[0453] The fourth feature data is subjected to data perturbation to obtain the fifth feature data;

[0454] Based on the fifth feature data, the third classification result of the candidate region is obtained;

[0455] The third classification loss is obtained based on the difference between the third classification result and the second category;

[0456] The target detection loss is obtained based on the second classification loss, the first regression loss, the first interest loss, the second interest loss, and the third classification loss.

[0457] In any embodiment of this application, the acquisition unit 11 is used for:

[0458] Based on the fifth feature data, the third position of the detected object in the second training image is obtained;

[0459] The second regression loss is obtained based on the difference between the first position and the third position;

[0460] The target detection loss is obtained based on the second classification loss, the first regression loss, the first interest loss, the second interest loss, the third classification loss, and the second regression loss.

[0461] In this embodiment, the image processing device uses a second training image to train the target detection network, enabling the network to learn target detection capabilities. At least one first interfering image is obtained by perturbing the first training image. Therefore, while training the deep learning network using the first training image, the target detection network can learn, during training, how to accurately extract feature information from the first training image without perturbing, and how to accurately extract feature information from the first interfering image after perturbing, thereby improving the classification accuracy of the target detection network. Furthermore, since the classification accuracy and target detection accuracy of the target detection network are positively correlated, improving the classification accuracy of the target detection network can improve its target detection accuracy.

[0462] In some embodiments, the functions or modules of the apparatus provided in this application can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0463] Figure 4 This is a schematic diagram of the hardware structure of an image processing device provided in an embodiment of this application. The image processing device 2 includes a processor 21, a memory 22, an input device 23, and an output device 24. The processor 21, memory 22, input device 23, and output device 24 are coupled together via connectors, which may include various interfaces, transmission lines, or buses, etc., and are not limited in this embodiment. It should be understood that in the various embodiments of this application, coupling refers to mutual connection through a specific method, including direct connection or indirect connection through other devices, such as through various interfaces, transmission lines, buses, etc.

[0464] Processor 21 can be one or more graphics processing units (GPUs). If processor 21 is a GPU, the GPU can be a single-core GPU or a multi-core GPU. Optionally, processor 21 can be a processor group composed of multiple GPUs, with the multiple processors coupled to each other via one or more buses. Optionally, the processor can also be other types of processors, etc., which are not limited in this embodiment.

[0465] The memory 22 can be used to store computer program instructions, as well as various types of computer program code, including program code for executing the scheme of this application. Optionally, the memory includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM), which is used for related instructions and data.

[0466] Input device 23 is used to input data and / or signals, and output device 24 is used to output data and / or signals. Input device 23 and output device 24 can be independent devices or an integrated device.

[0467] It is understood that in this embodiment of the application, the memory 22 can be used not only to store related instructions, but also to store related data. For example, the memory 22 can be used to store the image to be processed obtained through the input device 23, or the memory 22 can also be used to store the first detection result obtained through the processor 21, etc. This embodiment of the application does not limit the specific data stored in the memory.

[0468] Understandable, Figure 4 This is merely a simplified design of an image processing apparatus. In practical applications, the image processing apparatus may also include other necessary components, including, but not limited to, any number of input / output devices, processors, memories, etc., and all image processing apparatuses that can implement the embodiments of this application are within the protection scope of this application.

[0469] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can implement the described functions using different methods for each specific application, but such implementation should not be considered beyond the scope of this application.

[0470] Those skilled in the art will readily understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. Those skilled in the art will also readily understand that the various embodiments of this application have different focuses, and for the sake of convenience and brevity, the same or similar parts may not be repeated in different embodiments. Therefore, parts not described or not described in detail in one embodiment can be referred to the descriptions in other embodiments.

[0471] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0472] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0473] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0474] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., digital versatile discs (DVDs)), or semiconductor media (e.g., solid-state disks (SSDs)).

[0475] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. An image processing method, characterized in that, The method includes: The image to be processed and the target detection network are obtained. The target detection network is trained by a first training image, a second training image and at least one first interference image. The at least one first interference image is obtained by performing image interference processing on the first training image. The first annotation data of the first training image is the first category of the first training image. The second annotation data of the second training image includes the second category of the detected object in the second training image and the first position of the detected object in the second training image. The process of acquiring the target detection network includes: acquiring a network to be trained, a first training image, a second training image, and at least one first interference image; processing the first training image and the at least one first interference image using the network to be trained to obtain a robust loss; processing the second training image using the network to be trained to obtain a target detection loss; obtaining the total loss of the network to be trained based on the target detection loss and the robust loss; and updating the parameters of the network to be trained based on the total loss to obtain the target detection network. The step of processing the first training image and the at least one first interference image using the network to be trained to obtain a robust loss includes: processing the first training image using the network to be trained to obtain a first classification result of the first training image; obtaining a first classification loss based on the difference between the first classification result and the first category; processing the at least one first interference image using the network to be trained to obtain an interference loss; and performing a weighted summation of the first classification loss and the interference loss to obtain the robust loss. The target detection network is used to perform target detection on the image to be processed to obtain a first detection result.

2. The method according to claim 1, characterized in that, Obtaining the first training image includes: Based on the second annotation data, the pixel region including the detected object is extracted from the second training image to obtain the first training image.

3. The method according to claim 1 or 2, characterized in that, The at least one first interference image includes a second interference image and a third interference image that is different from the second interference image; The step of processing the at least one first interference image using the network to be trained to obtain interference loss includes: The network to be trained is used to perform feature extraction processing on the second interference image and the third interference image respectively to obtain the first feature data of the second interference image and the second feature data of the third interference image; The enhancement loss is obtained based on the difference between the first feature data and the second feature data; The interference loss is obtained based on the enhancement loss.

4. The method according to claim 3, characterized in that, The at least one first interference image includes a fourth interference image, which is obtained by rotating the first training image by a first angle. Before obtaining the interference loss based on the enhancement loss, the method further includes: The fourth interference image is processed using the network to be trained to obtain a second angle, which represents the rotation angle of the fourth interference image relative to the first training image. The rotational loss is obtained based on the difference between the first angle and the second angle; The process of obtaining the interference loss based on the enhancement loss includes: The interference loss is obtained based on the enhancement loss and rotation loss.

5. The method according to claim 1 or 2, characterized in that, The step of processing the second training image using the network to be trained to obtain the target detection loss includes: The second training image is processed using the network to be trained to obtain a second classification result of the detected object and a second position of the detected object in the second training image; The second classification loss is obtained based on the difference between the second category and the second classification result; The first regression loss is obtained based on the difference between the first position and the second position; The target detection loss is obtained based on the second classification loss and the first regression loss.

6. The method according to claim 5, characterized in that, The step of processing the second training image using the network to be trained to obtain the second classification result of the detected object and the second position of the detected object in the second training image includes: The second training image is processed using the network to be trained to determine the candidate region of the detection object from the second training image; Determine the category of the candidate region to obtain the second classification result; The position of the candidate region in the second training image is determined to obtain the second position.

7. The method according to claim 6, characterized in that, Before obtaining the target detection loss based on the second classification loss and the first regression loss, the method further includes: The probability of the candidate region being a foreground region is determined to obtain a second detection result, wherein the foreground region is the pixel region corresponding to the detected object; Based on the second detection result, the first position, and the second position, a first interest loss is obtained, and the first interest loss is negatively correlated with the accuracy of the second detection result; The method of obtaining the target detection loss based on the second classification loss and the first regression loss includes: The target detection loss is obtained based on the second classification loss, the first regression loss, and the first interest loss.

8. The method according to claim 7, characterized in that, Before obtaining the target detection loss based on the second classification loss, the first regression loss, and the first interest loss, the method further includes: The first training image is processed by feature extraction using the network to be trained to obtain the third feature data of the first training image; Based on the third feature data, the probability that the first training image is the foreground region is determined, and a third detection result is obtained; Based on the difference between the third detection result and the first labeled data, a second loss of interest is obtained; The target detection loss is obtained based on the second classification loss, the first regression loss, and the first interest loss, including: The target detection loss is obtained based on the second classification loss, the first regression loss, the first interest loss, and the second interest loss.

9. The method according to claim 8, characterized in that, Before obtaining the target detection loss based on the second classification loss, the first regression loss, the first interest loss, and the second interest loss, the method further includes: The candidate region is processed by the network to be trained to obtain the fourth feature data of the candidate region; The fourth feature data is subjected to data perturbation to obtain the fifth feature data; Based on the fifth feature data, the third classification result of the candidate region is obtained; The third classification loss is obtained based on the difference between the third classification result and the second category; The target detection loss is obtained based on the second classification loss, the first regression loss, the first interest loss, and the second interest loss, including: The target detection loss is obtained based on the second classification loss, the first regression loss, the first interest loss, the second interest loss, and the third classification loss.

10. The method according to claim 9, characterized in that, Before obtaining the target detection loss based on the second classification loss, the first regression loss, the first interest loss, the second interest loss, and the third classification loss, the method further includes: Based on the fifth feature data, the third position of the detected object in the second training image is obtained; The second regression loss is obtained based on the difference between the first position and the third position; The target detection loss is obtained based on the second classification loss, the first regression loss, the first interest loss, the second interest loss, and the third classification loss, including: The target detection loss is obtained based on the second classification loss, the first regression loss, the first interest loss, the second interest loss, the third classification loss, and the second regression loss.

11. An image processing apparatus, characterized in that, The device includes: An acquisition unit is used to acquire an image to be processed and an object detection network. The object detection network is trained using a first training image, a second training image, and at least one first interference image. The at least one first interference image is obtained by performing image interference processing on the first training image. The first annotation data of the first training image is the first category of the first training image. The second annotation data of the second training image includes the second category of the detected object in the second training image and the first position of the detected object in the second training image. The process of acquiring the target detection network includes: acquiring a network to be trained, a first training image, a second training image, and at least one first interference image; processing the first training image and the at least one first interference image using the network to be trained to obtain a robust loss; processing the second training image using the network to be trained to obtain a target detection loss; obtaining the total loss of the network to be trained based on the target detection loss and the robust loss; and updating the parameters of the network to be trained based on the total loss to obtain the target detection network. The step of processing the first training image and the at least one first interference image using the network to be trained to obtain a robust loss includes: processing the first training image using the network to be trained to obtain a first classification result of the first training image; obtaining a first classification loss based on the difference between the first classification result and the first category; processing the at least one first interference image using the network to be trained to obtain an interference loss; and performing a weighted summation of the first classification loss and the interference loss to obtain the robust loss. The processing unit is used to perform target detection on the image to be processed using the target detection network to obtain a first detection result.

12. An electronic device, characterized in that, include: A processor and a memory, the memory being used to store computer program code, the computer program code including computer instructions, wherein, when the processor executes the computer instructions, the electronic device performs the method as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 10.

Images