Image processing and target detection method, device, equipment and storage medium

Abstract

The present disclosure provides an image processing and target detection method, device, equipment and storage medium, wherein the method comprises: acquiring an original image; performing N times of enhancement processing on the original image respectively to obtain N corresponding enhanced images; for each enhanced image in the N enhanced images, using a backbone network in a neural network to perform feature extraction on the enhanced image to obtain global image features of the enhanced image, and determining local image features for representing semantic objects in the enhanced image based on the global image features of the enhanced image; and adjusting the neural network based on the global image features and the local image features determined for each enhanced image respectively. The present disclosure combines the global image features and the local image features to adjust the network, which better narrows the features of different enhanced images and the features of different enhanced images for the same semantic objects. The self-supervised learning performed under this guidance has better training performance.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of artificial intelligence, and in particular, to an image processing and target detection method and device, equipment and a storage medium. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, end-to-end deep learning technology is also becoming mature and is widely used in various fields such as target detection, image classification, etc. Among them, supervised learning has achieved great success in various application fields.

[0003] However, supervised learning often requires a large number of artificial label samples, and generally it is very difficult to obtain artificial labels, and the quality of the labels is difficult to guarantee. Self-supervised learning aims to learn the representation directly from the structure of the data itself without artificial labels, and has more extensive practical significance.

[0004] Generally, the self-supervised learning method can use the contrast learning method, that is, the similarity of the global features of two perspectives of the same image can be used to construct the loss function of the self-supervised sample.

[0005] However, due to the introduction of object-independent background noise and spatial misalignment (i.e., the same object cannot be in the same spatial position under different perspectives), this leads to great problems in the process of downstream task application. SUMMARY

[0006] The embodiments of the present disclosure at least provide an image processing and target detection method, device, equipment and storage medium.

[0007] In a first aspect, the embodiments of the present disclosure provide an image processing method, comprising:

[0008] obtaining an original image;

[0009] performing N times of enhancement processing on the original image respectively to obtain N corresponding enhanced images, N≥2;

[0010] For each enhanced image in the N enhanced images, a backbone network in a neural network is used to extract features of the enhanced image to obtain global image features of the enhanced image, and local image features of a semantic object in the enhanced image are determined based on the global image features of the enhanced image;

[0011] Based on the similarity relationship between the global image features of the N enhanced images corresponding to at least one original image, and the similarity relationship between the local image features of the N enhanced images corresponding to at least one original image, the neural network is adjusted.

[0012] With the image processing method, in the case of obtaining the original image, the original image can be processed multiple times to obtain N enhanced images. The enhanced images retain the local or overall image features of the original image to some extent. Thus, for each enhanced image, the backbone network included in the neural network to be trained can be used to extract the global image features of the enhanced image, and the local image features can be determined based on the global image. Then, the network can be adjusted based on the global image features and the local image features.

[0013] The present disclosure combines the global image features for expressing overall image features and the local image features for expressing aligned target image features to adjust the network. In the premise of better narrowing the features of different enhanced images, the features of the same semantic object in different enhanced images are also well narrowed. The self-supervised learning performed in this guidance has better training performance, thereby ensuring the application in subsequent downstream tasks.

[0014] In a possible implementation, the local image features for characterizing the semantic objects in the enhanced image are determined based on the global image features of the enhanced image, including:

[0015] The spatial feature map of the global image features of the enhanced image is obtained by spatial feature extraction. Each channel of the spatial feature map corresponds to a semantic object, and each channel of the spatial feature map contains a probability map representing the predicted probability of each pixel belonging to the corresponding semantic object.

[0016] The local image features for characterizing the semantic objects in the enhanced image are determined based on the spatial feature map and the global image features.

[0017] Here, the spatial feature map representing the predicted probability of each semantic object is first determined, and then a series of local image features are obtained by spatial aggregation. The local image features of the same semantic object in different enhanced images can better evaluate the detection accuracy of the semantic object, thereby ensuring the accuracy of subsequent network adjustment.

[0018] In a possible implementation, the local image features for characterizing the semantic objects in the enhanced image are determined based on the spatial feature map and the global image features, including:

[0019] For each semantic object, the local image features for characterizing the semantic object are determined based on the probability map corresponding to the semantic object and the global image features.

[0020] In a possible implementation, the adjusting the neural network based on the similarity relationship between the global image features of the N enhanced images corresponding to at least one original image and the similarity relationship between the local image features of the N enhanced images corresponding to at least one original image comprises:

[0021] determining a first loss for the neural network based on the similarity relationship between the global image features of the N enhanced images corresponding to the single original image;

[0022] and determining a second loss for the neural network based on the similarity relationship between the local image features of the N enhanced images corresponding to the single original image;

[0023] adjusting the neural network at least one round based on the first loss and the second loss.

[0024] Here, the neural network can be adjusted one or more rounds in combination with the first loss and the second loss, and the neural network after adjustment is more and more close to a smaller loss, and thus is more and more close to better training performance.

[0025] In a possible implementation, the adjusting the neural network based on the similarity relationship between the global image features of the N enhanced images corresponding to at least one original image and the similarity relationship between the local image features of the N enhanced images corresponding to at least one original image further comprises:

[0026] determining a third loss for the neural network to be trained based on the similarity relationship between the global image features of the enhanced images corresponding to multiple original images;

[0027] determining a fourth loss for the neural network to be trained based on the similarity relationship between the local image features of the enhanced images corresponding to multiple original images;

[0028] The adjusting the neural network at least one round based on the first loss and the second loss comprises:

[0029] adjusting the neural network at least one round based on the first loss, the second loss, the third loss, and the fourth loss.

[0030] In a possible implementation, the first loss is determined in the following manner:

[0031] performing global average pooling operation on the global image features of the N enhanced images of the single original image to obtain global average image features corresponding to the N enhanced images respectively;

[0032] determine a first loss for the neural network based on similarity relationships between the global average image features respectively corresponding to the N enhanced images.

[0033] Here, the image features can be averaged based on the global average pooling layer, so that the determined global average image features can more globally and uniformly represent the enhanced images, further making the determined first loss better represent the closeness of the enhanced images at the global level.

[0034] In one possible implementation, each enhancement processing includes one or more of the following processing modes:

[0035] Random cropping, horizontal inversion, color disturbance, grayscale conversion, Gaussian blur.

[0036] In one possible implementation, after adjusting the neural network, the method further includes:

[0037] obtaining a target image and label information obtained by performing pixel-by-pixel semantic class labeling on the target image;

[0038] inputting the target image into the trained neural network to obtain detection information of the target object in the target image;

[0039] determining an image processing performance evaluation result of the neural network based on a comparison result of the detection information and the label information.

[0040] Here, the comparison result between the detection information output by the currently trained neural network and the pre-set label information can be used to verify the training performance of the network. The closer the two information, the better the network training performance to a certain extent. On the contrary, the farther the two information, the worse the network training performance to a certain extent. Based on this, more refined network adjustment can be achieved.

[0041] In a second aspect, the embodiments of the present disclosure also provide a target detection method, including:

[0042] obtaining a to-be-detected image;

[0043] inputting the to-be-detected image into the neural network adjusted for multiple rounds to extract global image features of the to-be-detected image by using a backbone network in the neural network, wherein the neural network is adjusted for the multiple rounds based on the image processing method of any one of the first aspect and various embodiments thereof;

[0044] determining a target detection result for the to-be-detected image based on the global image features.

[0045] In a third aspect, the embodiments of the present disclosure further provide an image processing apparatus, comprising:

[0046] an acquisition module, configured to acquire an original image;

[0047] a processing module, configured to perform N times of enhancement processing on the original image respectively to obtain N enhanced images, where N is greater than or equal to 2;

[0048] a determination module, configured to, for each of the N enhanced images, perform feature extraction on the enhanced image by using a backbone network in a neural network to obtain global image features of the enhanced image, and determine local image features for representing semantic objects in the enhanced image based on the global image features of the enhanced image;

[0049] an adjustment module, configured to adjust the neural network based on a similarity relationship between the global image features of the N enhanced images corresponding to at least one original image, and a similarity relationship between the local image features of the N enhanced images corresponding to at least one original image.

[0050] In a fourth aspect, the embodiments of the present disclosure further provide a target detection apparatus, comprising:

[0051] an acquisition module, configured to acquire a to-be-detected image;

[0052] a determination module, configured to input the to-be-detected image into a neural network that has been adjusted for multiple rounds to extract global image features of the to-be-detected image by using a backbone network in the neural network, where the neural network is adjusted for the multiple rounds based on the image processing method according to any one of the first aspect and various embodiments thereof;

[0053] a detection module, configured to determine a target detection result for the to-be-detected image based on the global image features.

[0054] In a fifth aspect, the embodiments of the present disclosure further provide an electronic device, comprising a processor, a memory and a bus, the memory stores machine readable instructions executable by the processor, when the electronic device is running, the processor and the memory communicate through the bus, and the machine readable instructions are executed by the processor to perform the steps of the image processing method according to any one of the first aspect and various embodiments thereof or the steps of the target detection method according to the second aspect.

[0055] In a sixth aspect, the present disclosure also provides a computer-readable storage medium, which stores a computer program. When the computer program is run by a processor, the steps of the image processing method according to any one of the first aspect and various embodiments thereof or the steps of the target detection method according to the second aspect are performed.

[0056] The effects of the above-described apparatus, electronic device, and computer-readable storage medium are described in the description of the above-described method, and will not be repeated here.

[0057] In order to make the above objectives, characteristics and advantages of the present disclosure more obvious and easy to understand, the following will specifically describe preferred embodiments, and combine with the accompanying drawings, make a detailed description as follows. BRIEF DESCRIPTION OF DRAWINGS

[0058] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following will briefly introduce the drawings needed to be used in the embodiments, the drawings herein are incorporated into the description and form a part of the description, the drawings show the embodiments consistent with the present disclosure, and are used to illustrate the technical solutions of the present disclosure together with the description. It should be understood that the following drawings only show some embodiments of the present disclosure, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0059] Figure 1 A flow chart of an image processing method provided by an embodiment of the present disclosure is shown;

[0060] Figure 2 An application schematic diagram of an image processing method provided by an embodiment of the present disclosure is shown;

[0061] Figure 3 A flow chart of a target detection method provided by an embodiment of the present disclosure is shown;

[0062] Figure 4 A schematic diagram of an image processing apparatus provided by an embodiment of the present disclosure is shown;

[0063] Figure 5 A schematic diagram of a target detection apparatus provided by an embodiment of the present disclosure is shown;

[0064] Figure 6 A schematic diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0065] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, and not all of them. The components of the embodiments of this disclosure described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed disclosure, but merely represents selected embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without inventive effort are within the scope of protection of this disclosure.

[0066] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0067] In this document, the term "and / or" merely describes a relationship, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0068] Research has shown that self-supervised learning methods can employ a contrastive learning approach, meaning that the loss function for self-supervised samples can be constructed based on the similarity of global features from two perspectives of the same image.

[0069] However, the introduction of object-independent background noise and spatial misalignment (i.e., the same object cannot be in the same spatial position from different viewpoints) causes significant problems in downstream task applications.

[0070] Based on the above research, this disclosure provides at least one scheme for training a network by combining global image features and local image features, so as to improve the performance of network training and thus ensure its application in subsequent downstream tasks.

[0071] For the convenience of understanding the present embodiment, first, a kind of image processing method disclosed in the present embodiment is introduced in detail, the execution subject of the image processing method provided by the present embodiment is generally electronic equipment with certain computing power, which includes, for example: terminal equipment or server or other processing equipment, terminal equipment can be user equipment (User Equipment, UE), mobile device, user terminal, cellular phone, cordless phone, personal digital assistant (Personal Digital Assistant, PDA), handheld device, wearable device, etc.In some possible implementation ways, the image processing method can be realized by the way that processor calls computer readable instructions stored in memory.

[0072] Referring to Figure 1 As shown in the flowchart of the image processing method provided by the present embodiment, the method comprises steps S101-S104, wherein:

[0073] S101: obtaining an original image;

[0074] S102: performing N times of enhancement processing on the original image respectively, to obtain corresponding N enhanced images, N≥2;

[0075] S103: for each enhanced image in the N enhanced images, the backbone network in the neural network is used to extract features of the enhanced image, to obtain global image features of the enhanced image, and based on the global image features of the enhanced image, local image features for representing semantic objects in the enhanced image are determined;

[0076] S104: based on the similarity relationship between the global image features of the N enhanced images corresponding to at least one original image, and the similarity relationship between the local image features of the N enhanced images corresponding to at least one original image, the neural network is adjusted.

[0077] In order to facilitate understanding of the image processing method provided by the present embodiment, next, first, the application scenario of the method is simply explained.The image processing method in the present embodiment can be mainly applied to the field of vision, for example, based on the trained neural network can be applied to target detection, image classification and other task scenarios.

[0078] In view of the method of self-supervised learning in the related art by using the contrast learning manner, due to the introduction of object-independent background noise and spatial misalignment, the related training scheme of constructing the loss function of the self-supervised sample based on the similarity of the global features of two views of the same image cannot achieve better training performance. In order to solve this problem, the network training scheme for consistency constraint of global image features and local image features is provided in the embodiments of the present disclosure to improve the network training performance in self-supervised learning.

[0079] Here, the global image features and the local image features determined based on each enhanced image can be used to adjust the neural network, that is, in the process of adjusting the neural network, not only the global image features of the enhanced image are referred to, but also the local image features of the enhanced image are referred to. The global image features are used to evaluate the enhanced image from the overall level, and the local image features can be used to evaluate the enhanced image from the local semantic object level. Coupling the learning of the two kinds of features can improve the performance of image-level prediction (corresponding to target prediction) and dense-level prediction (corresponding to pixel point prediction) at the same time, so that the network performance of the trained neural network can be significantly improved.

[0080] Among them, the enhanced image can be obtained based on N times of enhancement processing of the original image, for example, two enhanced images of the original image can be obtained by performing two times of data enhancement on the original image. In order to facilitate the description, the following examples can be performed twice.

[0081] The enhancement processing here can include various image processing methods such as random cropping, horizontal inversion, color disturbance, grayscale conversion, Gaussian blur, etc. In addition, other image processing methods can also be included, and the embodiments of the present disclosure do not make specific limitations thereto.

[0082] In actual application, the N times of enhancement processing in the embodiments of the present disclosure can be obtained by combining different enhancement processing methods, for example, one of the enhanced images can be obtained by performing horizontal inversion and color disturbance on the original image, and the other enhanced image can be obtained by performing random cropping and grayscale conversion on the original image.

[0083] It should be noted that the reason why the embodiments of the present disclosure adopt the way of random cropping and other enhancement processing to obtain the enhanced image is mainly to improve the training performance of self-supervised learning, so as to be able to mine more features from two views of the same image which are beneficial to subsequent task execution such as classification. However, the random operation described above causes a problem of spatial misalignment to some extent. In order to solve such a problem, the embodiments of the present disclosure provide a scheme of assisting global image features with local image features for self-supervised learning, which utilizes local image features to align different enhanced images in space, thereby greatly improving the application of subsequent tasks.

[0084] In the embodiments of the present disclosure, the corresponding global image features and local image features can be determined for each enhanced image. The global image features are used to represent deep features of the whole enhanced image, which can be extracted by using the backbone network in the neural network. The local image features can represent the local features of the semantic objects in the enhanced image, which can be obtained by spatial aggregation of the spatial feature maps generated by the multilayer perceptron included in the neural network.

[0085] Here, the loss of the neural network can be determined based on the global image features respectively determined for the N enhanced images and the local image features respectively determined for the N enhanced images, and then one or more rounds of parameter adjustment of the neural network are realized through the loss.

[0086] In the actual parameter adjustment process, the similarity relationship between the global image features of the N enhanced images corresponding to at least one original image and the similarity relationship between the local image features of the N enhanced images corresponding to at least one original image can be determined first, and then the network loss is determined based on the two similarity relationships, and then the parameter adjustment is performed.

[0087] Among them, the closer the global image features of the N enhanced images corresponding to one original image are, the closer the overall representation of the N enhanced images is to some extent, so the loss determined is smaller, on the contrary, the farther the global image features of the N enhanced images are, the farther the overall representation of the N enhanced images is to some extent, so the loss determined is larger; in addition, the closer the local image features of the N enhanced images corresponding to one original image are, the closer the local representation of the specific semantic objects in the N enhanced images is to some extent, so the loss determined is smaller, on the contrary, the farther the local image features of the N enhanced images are, the farther the local representation of the specific semantic objects in the N enhanced images is to some extent, so the loss determined is larger.

[0088] Based on the above description, the training goal that N enhanced images are close to each other in global image features and local image features can be achieved by using the constraint of loss.

[0089] Here, the loss determined by using two similarity relationships can be used to adjust the parameters of the neural network, which can be achieved by the following steps:

[0090] Step one, based on the similarity relationship between the global image features of the N enhanced images corresponding to a single original image, determine the first loss for the neural network; and based on the similarity relationship between the local image features of the N enhanced images corresponding to a single original image, determine the second loss for the neural network;

[0091] Step two, based on the first loss and the second loss, at least one round of parameter adjustment is performed on the neural network.

[0092] Here, the first loss corresponding to the global image features of the N enhanced images can be determined first, and the second loss corresponding to the local image features of the N enhanced images can be determined, and finally the adjustment of the network is realized based on the loss function and the value of the two losses (i.e. the first loss and the second loss).

[0093] The first loss can be determined based on the similarity relationship between the two global average image features output by the global average pooling layer when the global image features of the N enhanced images are input into the trained global average pooling layer. The higher the similarity relationship, the closer the two global features, and the smaller the first loss determined. On the contrary, the lower the similarity relationship, the farther the two global features, and the larger the first loss determined. Compared with global image features, global average image features can be the result of important feature extraction on multi-dimensional global image features.

[0094] In addition, the second loss can be determined based on the similarity relationship between the plurality of local image features. The higher the similarity relationship, the closer the plurality of local image features, and the smaller the second loss determined. On the contrary, the lower the similarity relationship, the farther the plurality of local image features, and the larger the second loss determined.

[0095] In the embodiment of the present disclosure, in the case that the current round of network training does not satisfy the network training iteration stop condition, the network parameter values of at least one of the backbone network and the multi-layer perceptron included in the neural network are adjusted based on the loss and value determined by the first loss and the second loss, to obtain an adjusted neural network, and the next round of network training is performed according to the adjusted neural network, until the network training iteration stop condition is satisfied.

[0096] It is known that the above network adjustment process is a process of iterative loop. As long as the current round of network training does not meet the network training iteration termination condition, the neural network can be adjusted based on the loss value, after adjusting the neural network, the next round of network training can be performed, at this time the adjusted loss is obtained, and then it can be determined whether to adjust the network again based on the condition judgment result, until the converged neural network is obtained.

[0097] The network training iteration termination condition can refer to that the training number reaches a preset number, the loss function value is less than a preset threshold, all original images are traversed, and the like, which is not limited specifically herein.

[0098] The first loss and the second loss in the embodiment of the disclosure are determined based on the feature similarity relationship between the multiple different enhanced images corresponding to the same original image, that is, more consideration is given to the intra-class loss, in addition, the parameter adjustment of the network can also be realized by combining the inter-class loss, which can be realized by the following steps:

[0099] Step one, determining a third loss for the neural network to be trained based on the similarity relationship between the global image features of the enhanced images corresponding to the multiple original images;

[0100] Step two, determining a fourth loss for the neural network to be trained based on the similarity relationship between the local image features of the enhanced images corresponding to the multiple original images;

[0101] Step three, performing at least one round of parameter adjustment on the neural network based on the first loss, the second loss, the third loss and the fourth loss.

[0102] Here, the third loss can be determined based on the similarity relationship between the global image features of the enhanced images corresponding to the multiple original images, and the fourth loss can be determined based on the similarity relationship between the local image features of the enhanced images corresponding to the multiple original images, and then the parameter adjustment is realized by combining the first loss and the second loss.

[0103] Corresponding to the first loss, the third loss here corresponds to the loss caused by the global feature similarity. Unlike the first loss which decreases with the improvement of the global feature similarity, the third loss here will increase with the improvement of the global feature similarity, that is, the purpose of adjusting the parameters here is to make the intra-class similarity high and the inter-class similarity low. Similarly, the adjustment of the fourth loss and the second loss is similar to the above process, which will not be described here.

[0104] Considering the key role of local image feature extraction in neural network adjustment, the determination process of local image features can be described in detail. The embodiment of the disclosure can determine the local image features of the enhanced image according to the following steps:

[0105] Step one, spatial feature extraction is performed on the global image features of the enhanced image to obtain a spatial feature map; each channel of the spatial feature map corresponds to a semantic object, and each channel of the spatial feature map contains a probability map representing the predicted probability of each pixel point belonging to the corresponding semantic object;

[0106] Step two, based on the spatial feature map and the global image features, local image features are determined to represent the semantic objects in the enhanced image.

[0107] Here, the multi-layer perceptron can be used to determine the spatial feature map representing the predicted probability of each pixel point belonging to each pre-set semantic object. The greater the predicted probability value on the corresponding pixel point, the greater the possibility that the pixel point is predicted to be the same semantic object. For multiple semantic objects, the multi-layer perceptron can correspondingly output a spatial feature map pointing to multiple probability maps, and each channel of the spatial feature map corresponds to predicting a semantic object.

[0108] Based on the spatial feature map and the global image features extracted in the early stage, a series of alignment features can be obtained by spatial aggregation, and each alignment feature corresponds to a local image feature indicating a semantic object.

[0109] In actual application, based on the multiplication operation between the probability map corresponding to the semantic object and the global image features, the local image features representing the semantic object are determined.

[0110] For the local image features of multiple enhanced images corresponding to an original image, feature matching can be performed based on the alignment features corresponding to the same semantic object, thereby realizing alignment in the spatial dimension. In this way, the corresponding second loss can be used to narrow the local features from the same semantic in the same image for self-supervised learning, further improving the learning performance.

[0111] In order to facilitate further understanding of the above network training process, the following can be combined Figure 2 for description.

[0112] As Figure 2 shown, an original image containing a cat is input, and after two enhancement processes of t' and t", two enhanced images (i.e., x' and x") can be obtained. For the two enhanced images, Encoderf θ and Encoderf ξIn the case that the two backbone networks extract corresponding global image features (i.e., F' and F''), on the one hand, the global average pooling layer GAP can be used for global averaging of the features to obtain corresponding global average image features (i.e., z' and z''), and the first loss (i.e., Global Loss L g ) can be determined based on the similarity relationship between z' and z''.

[0113] On the other hand, two multi-layer perceptrons (i.e., Coupled ProJg θ and Coupled ProJg ξ ) can be used to determine the spatial feature maps (i.e., W' and W'') generated for the two enhanced images, respectively, and W' and W'' correspond to a feature depth of d, which is used to represent d semantic objects. Based on this, in the case of multiplication operation of the global image features and the spatial feature maps, the local image features (i.e., {z' k} and {z'' k}) representing the semantic objects in the enhanced images can be obtained, and the second loss (i.e., Aligned Loss L a ) can be determined based on the similarity relationship between {z' k} and {z'' k}.

[0114] In the process of self-supervised learning, the network adjustment can be performed simultaneously by using the constraints of the above two losses, and the performance of the entire network training is better.

[0115] In the embodiments of the present disclosure, in the case that the trained neural network is obtained, the network performance of the neural network can be verified, which can be achieved by the following steps:

[0116] Step one, obtaining a target image and label information obtained by performing pixel-by-pixel semantic class labeling on the target image;

[0117] Step two, inputting the target image into the trained neural network to obtain detection information of the target object in the target image;

[0118] Step three, determining an image processing performance evaluation result of the neural network based on the comparison result of the detection information and the label information.

[0119] Here, the network performance can be verified based on a comparison result between the detection information of the target object in the target image obtained by the trained neural network and the pre-set annotation information. The more matched the detection information and the annotation information are, the stronger the processing performance of the trained neural network is to a certain extent, that is, the stronger the corresponding self-supervised learning performance is. On the contrary, in the case that the detection information and the annotation information are not matched, the processing performance of the trained neural network is weaker to a certain extent, that is, the self-supervised learning performance is weaker. In this case, the training of the neural network can be performed again until the neural network with higher network performance is obtained, which will help to improve the accuracy of subsequent task execution.

[0120] Based on the above image processing method provided by the embodiments of the present disclosure, the present disclosure also provides a target detection method, as shown in Figure 3 The method specifically comprises the following steps:

[0121] S301: acquiring a to-be-detected image;

[0122] S302: inputting the to-be-detected image into the neural network adjusted for multiple rounds to extract global image features of the to-be-detected image by using a backbone network in the neural network;

[0123] S303: determining a target detection result for the to-be-detected image based on the global image features.

[0124] The to-be-detected image acquired here can be an image captured by a collection device set in any target scene. After target detection is performed by using the trained neural network, a detection result including a category to which a target object in the image belongs can be obtained.

[0125] The training process of the neural network is described above and will not be repeated here.

[0126] Those skilled in the art can understand that the writing order of each step in the above method of the specific embodiment does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0127] Based on the same inventive concept, the embodiments of the present disclosure also provide a device corresponding to the method. Since the principle of solving problems by the device in the embodiments of the present disclosure is similar to the above-mentioned method, the implementation of the device can be referred to the implementation of the method, and the repeated parts will not be repeated.

[0128] Referring to Figure 4 As shown in FIG. 4, an image processing device provided by the embodiments of the present disclosure includes an acquisition module 401, a processing module 402, a determination module 403 and an adjustment module 404, wherein

[0129] The acquisition module 401 is configured to acquire an original image.

[0130] The processing module 402 is configured to perform N times of enhancement processing on the original image respectively to obtain N enhanced images, where N is greater than or equal to 2.

[0131] The determination module 403 is configured to, for each of the N enhanced images, perform feature extraction on the enhanced image by using a backbone network in the neural network to obtain global image features of the enhanced image, and determine local image features for representing semantic objects in the enhanced image based on the global image features of the enhanced image.

[0132] The adjustment module 404 is configured to adjust the neural network based on a similarity relationship between the global image features of the N enhanced images corresponding to at least one original image, and a similarity relationship between the local image features of the N enhanced images corresponding to the at least one original image.

[0133] The above image processing apparatus is combined with the global image features for representing overall image features and the local image features for representing aligned target image features for network adjustment, which better narrows down the features of different enhanced images and also better narrows down the features of the same semantic objects in different enhanced images. The self-supervised learning performed under this guidance has better training performance, thereby ensuring the application in subsequent downstream tasks.

[0134] In a possible implementation, the determination module 403 is configured to determine the local image features for representing the semantic objects in the enhanced image based on the global image features of the enhanced image according to the following steps:

[0135] perform spatial feature extraction on the global image features of the enhanced image to obtain a spatial feature map; each channel of the spatial feature map corresponds to a semantic object, and each channel of the spatial feature map contains a probability map representing a prediction probability that each pixel point belongs to the corresponding semantic object.

[0136] determine the local image features for representing the semantic objects in the enhanced image based on the spatial feature map and the global image features.

[0137] In a possible implementation, the determination module 403 is configured to determine the local image features for representing the semantic objects in the enhanced image based on the spatial feature map and the global image features according to the following steps:

[0138] for each semantic object, determine the local image features for representing the semantic object based on the probability map corresponding to the semantic object and the global image features.

[0139] In a possible implementation, the adjusting module 404 is configured to adjust the neural network according to the following steps:

[0140] determine a first loss for the neural network based on the similarity relationship between the global image features of the N enhanced images corresponding to the single original image, and determine a second loss for the neural network based on the similarity relationship between the local image features of the N enhanced images corresponding to the single original image;

[0141] perform at least one round of parameter adjustment on the neural network based on the first loss and the second loss, to obtain the trained neural network.

[0142] In a possible implementation, the adjusting module 404 is further configured to:

[0143] determine a third loss for the neural network to be trained based on the similarity relationship between the global image features of the enhanced images corresponding to the multiple original images;

[0144] determine a fourth loss for the neural network to be trained based on the similarity relationship between the local image features of the enhanced images corresponding to the multiple original images;

[0145] perform at least one round of parameter adjustment on the neural network based on the first loss, the second loss, the third loss, and the fourth loss.

[0146] In a possible implementation, the adjusting module 404 is configured to determine the first loss of the neural network according to the following steps:

[0147] perform global average pooling operation on the global image features of the N enhanced images of the single original image, to obtain global average image features corresponding to the N enhanced images respectively;

[0148] determine the first loss for the neural network based on the similarity relationship between the global average image features corresponding to the N enhanced images respectively.

[0149] In a possible implementation, each enhancement processing includes one or more of the following processing modes:

[0150] random cropping, horizontal inversion, color disturbance, grayscale conversion, and Gaussian blur.

[0151] In a possible implementation, the apparatus further includes a verifying module 405.

[0152] The verification module 405 is configured to, after adjusting the neural network, acquire a target image and label information obtained by performing pixel-by-pixel semantic category labeling on the target image; input the target image into the trained neural network to obtain detection information of a target object in the target image; and determine an image processing performance evaluation result of the neural network based on a comparison result of the detection information and the label information.

[0153] Referring to Figure 5 FIG. 1 is a schematic diagram of a target detection device provided by an embodiment of the present disclosure, which includes an acquisition module 501, a determination module 502, and a detection module 503.

[0154] The acquisition module 501 is configured to acquire a to-be-detected image.

[0155] The determination module 502 is configured to input the to-be-detected image into the neural network that has been adjusted for multiple rounds to extract global image features of the to-be-detected image by using a backbone network in the neural network.

[0156] The detection module 503 is configured to determine a target detection result for the to-be-detected image based on the global image features.

[0157] The processing procedure of each module in the device and the interaction procedure between the modules can be referred to the related descriptions in the above method embodiments, and will not be described in detail here.

[0158] Corresponding to the method in Figure 1 and Figure 3 , the present embodiment of the present disclosure further provides an electronic device, as shown in Figure 6 FIG. 2 is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure, which includes:

[0159] a processor 61, a memory 62, and a bus 63; the memory 62 is configured to store execution instructions, including an internal memory 621 and an external memory 622; the internal memory 621 is also referred to as an internal storage, and is configured to temporarily store operation data in the processor 61 and exchange data with the external memory 622 such as a hard disk; the processor 61 exchanges data with the external memory 622 through the internal memory 621, and when the electronic device 600 is running, the processor 61 and the memory 62 communicate through the bus 63, so that the processor 61 performs steps of an image processing method as shown in Figure 1 or steps of a target detection method as shown in Figure 3 .

[0160] The present embodiment of the present disclosure further provides a computer readable storage medium, which stores a computer program, and the computer program is run by a processor to perform the steps of the method described in the above method embodiments. The storage medium can be a volatile or non-volatile computer readable storage medium.

[0161] The embodiment of the disclosure further provides a computer program product carrying program codes, the program codes include instructions for executing the steps of the method described in the above method embodiments, and details can be referred to the above method embodiments, which will not be repeated here.

[0162] The computer program product can be specifically implemented by hardware, software or a combination thereof. In an optional embodiment, the computer program product is specifically embodied as a computer storage medium, and in another optional embodiment, the computer program product is specifically embodied as a software product, such as a software development kit (SDK) and the like.

[0163] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system and device can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here. In several embodiments provided by the disclosure, it should be understood that the disclosed system, device and method can be implemented by other ways. The above-described device embodiments are only schematic, for example, the division of the units is only a logical function division, and actual implementation can be in another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual coupling or direct coupling or communication connection can be indirect coupling or communication connection through some communication interfaces, devices or units, and can be electrical, mechanical or other forms.

[0164] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment.

[0165] In addition, each functional unit in each embodiment of the disclosure can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.

[0166] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a nonvolatile computer readable storage medium executable by a processor. Based on this understanding, the technical solutions of the present disclosure essentially or the part of the prior art that contributes to the present disclosure or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for making an electronic device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in various embodiments of the present disclosure. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (Read-Only Memory, ROM), a random access memory (Random Access Memory, RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0167] Finally, it should be noted that: the above-described embodiments are only specific embodiments of the present disclosure, used to illustrate the technical solutions of the present disclosure, and not to limit them. The protection scope of the present disclosure is not limited thereto. Although the present disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art who is familiar with the technical field of the present disclosure can make modifications or easily think of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed by the present disclosure, or make equivalent replacements to some of the technical features. The modifications, changes or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be covered within the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. An image processing method, characterized by, The method comprises: obtaining an original image; performing N times of enhancement processing on the original image respectively to obtain N corresponding enhanced images, N≥2; for each of the N enhanced images, performing feature extraction on the enhanced image by using a backbone network in a neural network to obtain global image features of the enhanced image, and determining local image features for representing semantic objects in the enhanced image based on the global image features of the enhanced image; adjusting the neural network based on similarity relationships between the global image features of the N enhanced images corresponding to at least one original image, and similarity relationships between the local image features of the N enhanced images corresponding to at least one original image.

2. The method of claim 1, wherein, The method further comprises: performing spatial feature extraction on the global image features of the enhanced image to obtain a spatial feature map; each channel of the spatial feature map corresponds to a semantic object, and each channel of the spatial feature map contains a probability map representing a prediction probability of each pixel belonging to the corresponding semantic object; determining the local image features for representing the semantic objects in the enhanced image based on the spatial feature map and the global image features.

3. The method of claim 2, wherein, The method further comprises: for each semantic object, determining the local image features for representing the semantic object based on the probability map corresponding to the semantic object and the global image features.

4. The method according to claim 2 or 3, characterized in that, The method further comprises: determining a first loss for the neural network based on the similarity relationships between the global image features of the N enhanced images corresponding to a single original image; determining a second loss for the neural network based on the similarity relationships between the local image features of the N enhanced images corresponding to a single original image; performing at least one round of parameter adjustment on the neural network based on the first loss and the second loss.

5. The method of claim 4, wherein, The method further comprises: determining a third loss for the neural network based on the similarity relationships between the global image features of the enhanced images corresponding to multiple original images; determining a fourth loss for the neural network based on the similarity relationships between the local image features of the enhanced images corresponding to multiple original images; The method further comprises: perform at least one round of parameter adjustment on the neural network based on the first loss, the second loss, the third loss, and the fourth loss.

6. The method according to claim 4 or 5, characterized in that, The first loss is determined in the following manner: perform global average pooling operation on global image features of the N enhanced images of the single original image to obtain global average image features corresponding to the N enhanced images respectively; determine a first loss for the neural network based on similarity relationships between the global average image features corresponding to the N enhanced images respectively.

7. The method according to any one of claims 1 to 6, characterized in that, Each enhancement processing includes one or more of the following: random cropping, horizontal inversion, color disturbance, grayscale conversion, Gaussian blur.

8. The method according to any one of claims 1 to 7, characterized in that, After adjusting the neural network, the method further includes: obtaining a target image and label information obtained by performing pixel-by-pixel semantic category labeling on the target image; inputting the target image into the trained neural network to obtain detection information of a target object in the target image; determining an image processing performance evaluation result of the neural network based on a comparison result of the detection information and the label information.

9. A target detection method characterized by, comprising: obtaining a to-be-detected image; inputting the to-be-detected image into a neural network adjusted for multiple rounds to extract global image features of the to-be-detected image by using a backbone network in the neural network, wherein the neural network is adjusted for the multiple rounds based on the image processing method of any one of claims 1 to 8; determining a target detection result for the to-be-detected image based on the global image features.

10. An image processing apparatus characterized by comprising: comprising: an obtaining module configured to obtain an original image; a processing module configured to perform N times of enhancement processing on the original image respectively to obtain corresponding N enhanced images, N≥2; a determining module configured to, for each enhanced image in the N enhanced images, extract features of the enhanced image by using a backbone network in a neural network to obtain global image features of the enhanced image, and determine local image features for representing semantic objects in the enhanced image based on the global image features of the enhanced image; an adjusting module configured to adjust the neural network based on similarity relationships between the global image features of the N enhanced images corresponding to at least one original image, and similarity relationships between the local image features of the N enhanced images corresponding to at least one original image.

11. A target detection apparatus characterized by comprising: comprising: an obtaining module configured to obtain a to-be-detected image; a determining module configured to input the to-be-detected image into a neural network adjusted for multiple rounds to extract global image features of the to-be-detected image by using a backbone network in the neural network, wherein the neural network is adjusted for the multiple rounds based on the image processing method of any one of claims 1 to 8; a detecting module configured to determine a target detection result for the to-be-detected image based on the global image features.

12. An electronic device, comprising: comprising: A processor, a memory, and a bus, the memory storing machine readable instructions executable by the processor, when the electronic device is running, the processor and the memory communicating through the bus, the machine readable instructions being executed by the processor to perform the steps of the image processing method of any one of claims 1 to 8 or the steps of the object detection method of claim 9.

13. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, the computer program being executed by the processor to perform the steps of the image processing method of any one of claims 1 to 8 or the steps of the object detection method of claim 9.

Images