Training method and device, image processing method, electronic device and storage medium
Through feature extraction and encoding processing of the image processing network, combined with unsupervised training methods, pseudo-segmentation labels are generated, which solves the problem that image segmentation is difficult to deal with multi-scale and unsupervised learning in the prior art, and achieves more efficient image segmentation accuracy and Accuracy.
Patent Information
- Application Number
- CN202111672616.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-12-31
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2041-12-31
AI Technical Summary
In the prior art, it is difficult to merge sub-regions with large appearance differences in the same object into a whole in image segmentation, and it is impossible to effectively process multi-scale situations and utilize the information of the entire data set, especially under unsupervised learning conditions, pixel-by-pixel Difficult to segment the label.
Using the training method of image processing network, by obtaining multiple pixel points arranged in the array, performing feature extraction and encoding processing, generating feature dictionary, using unsupervised training for image segmentation, combining encoded vectors and feature dictionary for clustering, generating Pseudo-segmentation tags are used to train image processing networks, reduce labeling workload and improve segmentation accuracy.
It improves the accuracy and efficiency of image segmentation, can better utilize the unmarked image data, reduces the workload of pixel-by-pixel labeling, and improves the segmentation accuracy of the image processing network, especially under multi-scale and unsupervised conditions.
Smart Images

Figure CN114330711B_ABST
Abstract
Description
Technical Field
[0001] Embodiments of the present disclosure relate to a training method for an image processing network, an image processing method, a training device for an image processing network, an image processing device, an electronic device, and a non-transitory computer-readable storage medium. Background Art
[0002] Image segmentation refers to the process of subdividing a digital image into multiple sub-regions. Depending on the definition of the sub-regions, it can be subdivided into semantic segmentation, instance segmentation, panoramic segmentation, etc. The related technologies of image segmentation have been widely used in industries such as autonomous driving, augmented reality, security monitoring, and medical auxiliary diagnosis. Image segmentation is an important basic task in computer vision and a classic problem. A universal method has not yet been found. In recent years, with the gradual deepening of deep learning technology, image segmentation algorithms have also made rapid progress. Summary of the invention
[0003] At least one embodiment of the present disclosure provides a method for training an image processing network, comprising: acquiring a training image, wherein the training image includes a plurality of pixel points arranged in an array; performing feature extraction processing on the training image using the image processing network to obtain a plurality of training feature vectors corresponding one-to-one to the plurality of pixel points; acquiring a feature dictionary, wherein the feature dictionary includes a plurality of image feature vectors; encoding the plurality of training feature vectors based on the plurality of image feature vectors to obtain a plurality of encoding vectors corresponding one-to-one to the plurality of training feature vectors; and performing image segmentation training on the image processing network according to the plurality of encoding vectors and the plurality of training feature vectors.
[0004] For example, in the training method provided in at least one embodiment of the present disclosure, the training image does not have a corresponding segmentation label, and the training performed on the image processing network is unsupervised training.
[0005] For example, in the training method provided in at least one embodiment of the present disclosure, obtaining a feature dictionary includes: obtaining a feature dictionary corresponding to a pre-trained image processing network as the feature dictionary.
[0006] For example, in the training method provided in at least one embodiment of the present disclosure, obtaining a feature dictionary includes: clustering the multiple pixel points according to the multiple training feature vectors to obtain K representative feature vectors, wherein the multiple image feature vectors include the K representative feature vectors, and K is a positive integer.
[0007] For example, in the training method provided by at least one embodiment of the present disclosure, the multiple pixel points are clustered according to the multiple training feature vectors to obtain K representative feature vectors, including: according to the multiple training feature vectors, the multiple pixel points are clustered or hierarchical clustered to divide the multiple pixel points into K first clusters; determining the centers of the K first clusters; and using the K training feature vectors corresponding to the centers of the K first clusters as the K representative feature vectors.
[0008] For example, in the training method provided by at least one embodiment of the present disclosure, obtaining a feature dictionary further includes: obtaining K historical representative feature vectors; wherein the multiple image feature vectors also include the K historical representative feature vectors.
[0009] For example, in the training method provided in at least one embodiment of the present disclosure, after obtaining a plurality of encoding vectors corresponding one-to-one to the plurality of training feature vectors, the method further includes: updating the feature dictionary.
[0010] For example, in the training method provided by at least one embodiment of the present disclosure, updating the feature dictionary includes: clustering the multiple pixel points according to the multiple encoding vectors to obtain N second clusters, wherein N is a positive integer greater than or equal to K; determining the total number of pixel points included in each second cluster to obtain N number of groups; selecting K second clusters from the N second clusters according to the N number of groups, wherein the number of groups of the K second clusters is greater than or equal to the number of groups of NK second clusters other than the K second clusters in the N second clusters; determining the centers of the K second clusters; and using the K training feature vectors corresponding to the centers of the K second clusters as the K historical representative feature vectors for the next round of training.
[0011] For example, in the training method provided by at least one embodiment of the present disclosure, N=2K.
[0012] For example, in the training method provided by at least one embodiment of the present disclosure, the multiple feature vectors are encoded based on the feature dictionary to obtain multiple encoding vectors corresponding to the multiple training feature vectors, including: for the i-th training feature vector among the multiple training feature vectors: performing similarity calculations on the i-th training feature vector and the multiple image feature vectors respectively to obtain multiple similarity values corresponding to the i-th training feature vector; using the multiple similarity values as the encoding vector corresponding to the i-th training feature vector, wherein i is a positive integer and is less than or equal to the total number of the multiple training feature vectors.
[0013] For example, in the training method provided in at least one embodiment of the present disclosure, the image processing network is trained according to the multiple encoding vectors and the multiple training feature vectors, including: clustering the multiple pixel points according to the multiple encoding vectors to obtain pseudo-segmentation labels corresponding to the training images; training the image processing network according to the pseudo-segmentation labels and the multiple training feature vectors.
[0014] For example, in the training method provided by at least one embodiment of the present disclosure, the multiple pixel points are clustered according to the multiple encoding vectors to obtain the pseudo-segmentation labels corresponding to the training image, including: clustering the multiple pixel points according to the multiple encoding vectors to obtain M third clusters, where M is a positive integer; marking the multiple pixel points according to the M third clusters to obtain the pseudo-segmentation labels, wherein the pixel points included in each third cluster are marked as belonging to the same category.
[0015] For example, in the training method provided by at least one embodiment of the present disclosure, the image processing network is trained according to the pseudo-segmentation label and the multiple training feature vectors, including: calculating the loss value of the image processing network according to the pseudo-segmentation label and the multiple training feature vectors through the loss function corresponding to the image processing network; and correcting the parameters of the image processing network based on the loss value; when the loss value corresponding to the image processing network does not meet a predetermined condition, continue to input the training image to repeat the above training process.
[0016] For example, in the training method provided in at least one embodiment of the present disclosure, obtaining a training image includes: obtaining an original image; performing multi-scale processing on the original image to obtain a plurality of processed images with different clarity; and performing connection processing on the plurality of processed images to obtain the training image.
[0017] For example, in the training method provided in at least one embodiment of the present disclosure, the original image is subjected to multi-scale processing to obtain a plurality of processed images with different clarity, including: obtaining a plurality of scaling ratios; performing size scaling processing on the original image according to the plurality of scaling ratios to obtain a plurality of intermediate images corresponding to the plurality of scaling ratios; performing size restoration processing on the plurality of intermediate images to obtain the plurality of processed images, wherein the sizes of the plurality of processed images are the same as the size of the original image.
[0018] For example, in the training method provided in at least one embodiment of the present disclosure, obtaining training images includes: obtaining a training image data set, wherein the training image data set includes multiple original images of the same type; traversing all original images in the training image data set, and using all the original images in turn as the training images.
[0019] At least one embodiment of the present disclosure provides an image processing method, comprising: acquiring an input image; performing image segmentation processing on the input image using an image processing network to obtain a segmented image corresponding to the input image, wherein the image processing network is at least partially trained according to the training method according to any one of claims 1-16.
[0020] For example, in at least one embodiment of the present disclosure, there is provided an image processing method, wherein the image segmentation processing includes semantic segmentation processing.
[0021] At least one embodiment of the present disclosure provides a training device for an image processing network, comprising: a training image acquisition unit, configured to acquire a training image, wherein the training image includes a plurality of pixel points arranged in an array; a feature extraction unit, configured to perform feature extraction processing on the training image using the image processing network, and obtain a plurality of training feature vectors corresponding one-to-one to the plurality of pixel points; a feature dictionary acquisition unit, configured to acquire a feature dictionary, wherein the feature dictionary includes a plurality of image feature vectors; an encoding processing unit, configured to perform encoding processing on the plurality of training feature vectors based on the plurality of image feature vectors, and obtain a plurality of encoding vectors corresponding one-to-one to the plurality of training feature vectors; and a training unit, configured to perform image segmentation training on the image processing network according to the plurality of encoding vectors and the plurality of training feature vectors.
[0022] At least one embodiment of the present disclosure provides an image processing device, comprising: an input image acquisition unit, configured to acquire an input image; a processing unit, configured to perform image segmentation processing on the input image using an image processing network to obtain a segmented image corresponding to the input image, wherein the image processing network is at least partially trained according to the training method described in at least one embodiment of the present disclosure.
[0023] At least one embodiment of the present disclosure provides an electronic device, comprising: a memory, which non-transitorily stores computer-executable instructions; and a processor, which is configured to execute the computer-executable instructions, wherein the computer-executable instructions, when executed by the processor, implement the training method of the image processing network described in any embodiment of the present disclosure or the image processing method described in any embodiment of the present disclosure.
[0024] At least one embodiment of the present disclosure provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the training method of the image processing network described in any embodiment of the present disclosure or the image processing method described in any embodiment of the present disclosure is implemented. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present disclosure, but are not intended to limit the present disclosure.
[0026] Figure 1 A training flow chart of an unsupervised image segmentation algorithm based on superpixel constraints is shown;
[0027] Figure 2 A schematic flowchart of an image processing method provided by at least one embodiment of the present disclosure;
[0028] Figure 3 A training flow chart provided for at least one embodiment of the present disclosure;
[0029] Figure 4 A schematic diagram of a segmentation effect provided by an embodiment of the present disclosure;
[0030] Figure 5 A schematic flowchart of an image processing method provided by at least one embodiment of the present disclosure;
[0031] Fig. 6A A schematic block diagram of a training device provided in at least one embodiment of the present disclosure;
[0032] Figure 6B A schematic block diagram of an image processing device provided by at least one embodiment of the present disclosure;
[0033] Figure 7 A schematic diagram of an electronic device provided for at least one embodiment of the present disclosure;
[0034] Figure 8 A schematic diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure;
[0035] Fig. 9 A schematic diagram of a hardware environment provided for at least one embodiment of the present disclosure. DETAILED DESCRIPTION
[0036] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.
[0037] Unless otherwise defined, the technical terms or scientific terms used in the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly. In order to keep the following description of the embodiments of the present disclosure clear and concise, the present disclosure omits the detailed description of some known functions and known components.
[0038] Image segmentation algorithms based on deep learning require a large amount of data to drive them, but it is very difficult to perform pixel-by-pixel segmentation and annotation on large-scale image data. Since unsupervised learning does not require pre-annotation of data, the use of unsupervised learning algorithms can better utilize large amounts of unlabeled image data.
[0039] Figure 1 A training flowchart of an unsupervised image segmentation algorithm based on superpixel constraints is shown.
[0040] like Figure 1 As shown, during the training process, the input image I is first input into the neural network to generate the initial segmentation result l = f(I|θ). For example, the processing function of the neural network is as follows Figure 1 It is shown as f(·|θ), where “·” represents the input image of the neural network; then, the initial segmentation result l is optimized according to the superpixel constraint to obtain the optimized result l′; then, the weight θ of the neural network is updated using the optimized result l′; the above process is repeated until the difference between the initial segmentation result l and the optimized result l′ is less than a preset threshold.
[0041] This unsupervised image segmentation algorithm based on superpixel constraints cannot merge sub-regions with large appearance differences in the same object into a whole sub-region. Figure 1 As shown in FIG. 1 , in the optimized result l′ after superpixel constraint optimization, the top of the ship is not segmented into a sub-region with the hull.
[0042] In addition, in some specific examples, this segmentation algorithm is also unable to handle multi-scale situations, and the same object will be segmented into different regions in input images of different sizes. Moreover, this segmentation algorithm can only learn based on a single image and cannot make good use of the information of the entire dataset. For example, the dataset includes a dataset consisting of images of the same type, such as a dataset consisting of natural scene images, a dataset consisting of texture images, a dataset consisting of pathological images, etc.
[0043] At least one embodiment of the present disclosure provides a training method for an image processing network, an image processing method, a training device for an image processing network, an image processing device, an electronic device, and a non-transitory computer-readable storage medium. The training method for the image processing network comprises: obtaining a training image, wherein the training image comprises a plurality of pixel points arranged in an array; performing feature extraction processing on the training image using the image processing network to obtain a plurality of training feature vectors corresponding one-to-one to the plurality of pixel points; obtaining a feature dictionary, wherein the feature dictionary comprises a plurality of image feature vectors; encoding processing on a plurality of training feature vectors based on the plurality of image feature vectors to obtain a plurality of encoding vectors corresponding one-to-one to the plurality of training feature vectors; and performing image segmentation training on the image processing network according to the plurality of encoding vectors and the plurality of training feature vectors.
[0044] The image processing method provided by at least one embodiment of the present disclosure integrates and collects image features through a feature dictionary, and uses the image feature vector in the feature dictionary to encode the training feature vector during the segmentation process. The encoding vector obtained by the encoding process is then used to perform segmentation training on the image processing network. The encoding vector contains the global image information integrated and collected in the feature dictionary, so that the feature dictionary can provide more accurate guidance for the segmentation of each image, thereby improving the image segmentation accuracy.
[0045] The image processing method provided in the embodiment of the present disclosure can be applied to the image processing device provided in the embodiment of the present disclosure, and the image processing device can be configured on an electronic device. The electronic device can be a personal computer, a mobile terminal, etc., and the mobile terminal can be a hardware device such as a mobile phone or a tablet computer.
[0046] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings, but the present disclosure is not limited to these specific embodiments.
[0047] Figure 2A schematic flowchart of an image processing method provided for at least one embodiment of the present disclosure.
[0048] For example, the image processing network in at least one embodiment of the present disclosure is a neural network. For example, the neural network can be a convolutional neural network, a deep neural network, etc. For example, the structure of the neural network can be U-Net, FCN (Fully Convolutional Networks), etc. The present disclosure does not specifically limit the type and structure of the neural network.
[0049] For example, the trained neural network is able to perform semantic segmentation on the input image.
[0050] like Figure 2 As shown, the image processing method provided by at least one embodiment of the present disclosure includes steps S10 to S50.
[0051] Step S10, obtaining a training image.
[0052] For example, the training image includes a plurality of pixels arranged in an array. For example, the size of the training image is W×H, where W represents the width of the training image, and H represents the height of the training image, and the training image includes a total of W×H pixels.
[0053] Step S20, using an image processing network to perform feature extraction processing on the training image to obtain a plurality of training feature vectors corresponding to a plurality of pixel points.
[0054] The present disclosure does not impose any specific restrictions on the specific process of feature extraction processing, image features extracted by the image processing network, etc.
[0055] For example, multiple training feature vectors are represented as That is, the training feature vector x corresponding to each pixel can be obtained through the image processing network. n , n is 1, 2, ..., W×H, x n is a c-dimensional vector, where c and n are both positive integers.
[0056] Step S30, obtaining a feature dictionary.
[0057] For example, the feature dictionary includes a plurality of image feature vectors.
[0058] Step S40: encoding the multiple training feature vectors based on the multiple image feature vectors to obtain multiple encoding vectors corresponding to the multiple training feature vectors.
[0059] Step S50, performing image segmentation training on the image processing network according to the multiple encoding vectors and the multiple training feature vectors.
[0060] For example, the training images do not have corresponding segmentation labels, and the training of the image processing network is unsupervised training. That is to say, the training images in the present disclosure do not need to be segmented and labeled one by one. During the training process, the clustering results of the coding vectors obtained after the coding process are used as estimated "image labels", and the parameters of the image processing network are corrected by combining the training feature vectors and the estimated "image labels". This method can make better use of a large amount of unlabeled image data, greatly reduce the workload of pixel-by-pixel labeling, and further improve the image segmentation accuracy by combining the image features of a large amount of unlabeled image data integrated in the feature dictionary through coding processing.
[0061] For example, the image feature vector included in the feature dictionary integrates and collects the image features of the images previously processed by the image processing network and the image features of the current training image. These image features are reflected by the image feature vector and merged with the training feature vector in step S40 to obtain the encoding vector. Since the training feature vector contains the local features of the image and the image feature vector reflects the global features of the image, the obtained encoding vector contains the local features and global features of the image, and can be used as a more accurate estimated "image label" for training the image processing network.
[0062] For example, in some embodiments, step S30 may include: obtaining a feature dictionary corresponding to a pre-trained image processing network as the feature dictionary.
[0063] For example, some training images are used in advance to train the image processing network. For example, the training images are training images in a data set, such as a data set of texture images. During the training process, the feature dictionary corresponding to the image processing network continuously collects image features of the training images in the data set. The specific collection method can refer to the content described below. Thus, the feature dictionary corresponding to the pre-trained image processing network is obtained.
[0064] Afterwards, the training images in the data set can be used to continue training the image processing network. The training process no longer updates the feature dictionary, but only updates the parameters of the image processing network. Alternatively, the training images in the data set can be used to train an untrained image processing network, and the feature dictionary participates in the training of the untrained image processing network as the known image features in the data set.
[0065] It should be noted that “pre-trained” here means that the image processing network has been trained, but it does not only mean that the image processing network has been trained.
[0066] For example, in some embodiments, the prediction results of the "pre-trained" image processing network still do not meet the predetermined accuracy conditions. At this time, it is still necessary to continue to input training images in the same data set to adjust the parameters of the image processing network, but this process may not update the feature dictionary.
[0067] For example, in other embodiments, the "pre-trained" image processing network may be an already trained image processing network, and a feature dictionary corresponding to the image processing network is obtained. The feature dictionary is then used to perform the above-mentioned training on a new, untrained image processing network. During the training process, the parameters of the untrained image processing network are updated in combination with the feature dictionary, thereby making full use of the existing image feature information and speeding up the training process.
[0068] For example, in some other embodiments, the feature dictionary may be updated in real time, and the feature dictionary continuously collects image features of training images during the training process.
[0069] Step S30 may include: performing clustering processing on a plurality of pixel points according to a plurality of training feature vectors to obtain K representative feature vectors, wherein the plurality of image feature vectors include K representative feature vectors, and K is a positive integer.
[0070] For example, clustering multiple pixel points according to multiple training feature vectors to obtain K representative feature vectors may include: performing K-means clustering or hierarchical clustering processing on the multiple pixel points according to the multiple training feature vectors to divide the multiple pixel points into K first clusters; determining the centers of the K first clusters; and using the K training feature vectors corresponding to the centers of the K first clusters as K representative feature vectors.
[0071] For example, the center of each first cluster can be obtained by the average position or average weighted position of all pixels included in the first cluster. For example, the center of the first cluster can be the pixel at the average position or the average weighted position; or, if the center does not correspond to a certain pixel, a pixel near the average position or the average weighted position (for example, within a predetermined distance range) is selected as the center of the first cluster.
[0072] For example, the steps of K-means clustering process are as follows:
[0073] Step 1: First, define K initial centers. For example, the K initial centers may be K random pixels in the training image, or the K initial centers may be obtained by using an initialization center algorithm.
[0074] Step 2: Allocate the training feature vectors corresponding to each pixel to one of the K clusters according to the degree of proximity. For example, each training feature vector is assigned to the cluster with the initial center closest to them. The measure of "degree of proximity" here is similarity. For example, calculate the similarity between the training feature vector corresponding to each pixel and the training feature vector corresponding to the center (e.g., the initial center of gravity), and allocate according to these similarities, so as to divide the W*H pixels into K clusters.
[0075] Step 3, move the current center to the new center of each cluster. The new position of the center of each cluster is obtained by calculating the average position or average weighted position of all pixels in the cluster. For example, the center of each cluster is the pixel point among W*H pixels that is located at or closest to the average position or average weighted position.
[0076] Repeat steps 2 and 3 until the position of the center no longer changes significantly in each iteration, thereby obtaining K centers and K first clusters.
[0077] For example, clustering processing may also adopt other clustering processing methods, which are not limited in the present disclosure.
[0078] For example, K representative feature vectors can be expressed as: Here, p i represents the i-th representative feature vector, that is, the training feature vector corresponding to the center of the i-th first cluster among the K first clusters, for example, p i is a c-dimensional vector.
[0079] By clustering the training feature vectors, the K most representative image features of the current training image are obtained, that is, K representative feature vectors. These K representative feature vectors are used as part of the feature dictionary to fuse and record the image features in the current training image.
[0080] For example, step S30 may further include: obtaining K historical representative feature vectors; wherein the multiple image feature vectors further include K historical representative feature vectors.
[0081] Here, the historical representative feature vector represents the most representative image feature of the image previously processed by the image processing network. The historical representative feature vector includes the image features of the image previously processed by the image processing network. Therefore, the historical representative feature vector can include global image information.
[0082] For example, K historical representative feature vectors can be expressed as: Here, q i represents the i-th historical representative feature vector among the K historical representative feature vectors, for example, q i is a c-dimensional vector.
[0083] For example, the historical representative feature vector can be dynamically updated. After the image processing network obtains the image features in the current input training image, it can update the historical representative feature vector according to the image features in the current input training image. For updating the historical representative feature vector, please refer to the description of feature dictionary update in the following text.
[0084] For example, when the training image is input to the image processing network for the first time, since the image processing network has not processed other images, the feature dictionary includes K representative feature vectors at this time, that is, the number of image feature vectors is K, and the feature dictionary D = D1. For example, when the training image is input to the image processing network for the second time and thereafter, the feature dictionary includes K historical representative feature vectors in addition to the K representative feature vectors corresponding to the current training image. The K historical representative feature vectors contain image features of images processed before the current training image, that is, the number of image feature vectors is 2*K, and the feature dictionary D = concatenate (D1, D2), where concatenate means connection processing.
[0085] For example, step S40 may include: for the i-th training feature vector among multiple training feature vectors: performing similarity calculations on the i-th training feature vector and multiple image feature vectors respectively to obtain multiple similarity values corresponding to the i-th training feature vector; using the multiple similarity values as the encoding vector corresponding to the i-th training feature vector, where i is a positive integer and is less than or equal to the total number of the multiple training feature vectors.
[0086] For example, the encoding vector is defined as the training feature vector x corresponding to the i-th pixel i A vector consisting of the similarities with all image feature vectors in the feature dictionary.
[0087] For example, the training feature vector x i The corresponding encoding vector E(xi) is expressed as: here, Represents the training feature vector x i With the image feature vector y j The specific calculation formula is shown in Formula 1:
[0088]
[0089] Here, Dist(x i ,y j ) represents the training feature vector x i and the image feature vector y j The similarity between them, e represents the Euler number, sim(x i -yj ) represents the training feature vector x i and the image feature vector y j The cosine similarity between them is calculated as shown in Formula 2:
[0090]
[0091] Here, x and y are two vectors of equal length and length c, x i and i Respectively represent the values of the i-th position of the two vectors.
[0092] For example, the training feature vector x is calculated by formula 1 i and K or 2*K image feature vectors y j The similarity value between them is obtained, thus obtaining the training feature vector x i The corresponding K or 2*K similarity values constitute the training feature vector x i The corresponding encoding vector E(xi).
[0093] For example, when the training image is input into the image processing network for the first time, each encoding vector obtained in step S40 includes K similarity values, that is, each encoding vector is a one-dimensional vector with a length of K; for example, when the training image is input into the image processing network for the second time and thereafter, each encoding vector obtained in step S40 includes 2*K similarity values, that is, each encoding vector is a one-dimensional vector with a length of 2*K.
[0094] By encoding the training feature vector, the training feature vector is integrated with the image features of the current training image and the historical image features, thereby obtaining the "image label" estimated by the image processing network, and used as a reference value to train the image processing network. In addition, the training feature vector expresses more local features and appearance features of the image, such as color features, etc. The historical image features can be integrated through encoding processing, so that the obtained encoding vector reflects both the local features of the image and the global features of the image, and the obtained image features are more comprehensive.
[0095] For example, step S50 may include: performing clustering processing on multiple pixel points according to multiple encoding vectors to obtain pseudo segmentation labels corresponding to the training image; and training the image processing network according to the pseudo segmentation labels and multiple training feature vectors.
[0096] For example, clustering multiple pixel points according to multiple coding vectors to obtain pseudo-segmentation labels may include: clustering multiple pixel points according to multiple coding vectors to obtain M third clusters, where M is a positive integer; marking multiple pixel points according to the M third clusters to obtain pseudo-segmentation labels, wherein the pixels included in each third cluster are marked as belonging to the same category.
[0097] For example, clustering multiple pixel points according to multiple coding vectors to obtain M third clusters may include: performing K-means clustering or hierarchical clustering processing on the multiple pixel points according to the multiple coding vectors to divide the multiple pixel points into M third clusters.
[0098] For example, the clustering process here can refer to the relevant content of the first clustering mentioned above, which will not be repeated here.
[0099] For example, clustering processing may also adopt other clustering processing methods, which are not limited in the present disclosure.
[0100] It should be noted that the processing flow or processing method of the clustering processing of the third clustering may be the same as or different from the clustering processing method of the first clustering, and the present disclosure does not impose any limitation on this.
[0101] For example, pixels belonging to the same cluster are labeled with the same number, so that each pixel in the training image is labeled with a different number according to the cluster it belongs to, thereby achieving automatic image segmentation and obtaining the "image label" estimated by the image processing network, that is, the pseudo-segmentation label.
[0102] For example, M may be equal to K or may not be equal to K.
[0103] For example, training an image processing network based on a pseudo-segmentation label and a plurality of training feature vectors may include: calculating a loss value of the image processing network through a loss function corresponding to the image processing network based on the pseudo-segmentation label and the plurality of training feature vectors; and correcting parameters of the image processing network based on the loss value; and when the loss value corresponding to the image processing network does not meet a predetermined condition, continuing to input training images to repeat the above training process.
[0104] For example, the pseudo-segmentation label is used as the image segmentation label estimated by the image processing network, and the pseudo-segmentation label is compared with multiple training feature vectors, and the parameters of the image processing network are corrected according to the comparison result. Specifically, the loss value of the image processing network can be calculated according to the pseudo-segmentation label and multiple training feature vectors using the loss function corresponding to the image processing network. For example, the loss function corresponding to the image processing network can be a cross entropy loss function; then, if the loss value does not meet the predetermined conditions, for example, the prediction result of the image processing network does not meet the predetermined accuracy condition, the parameters of the image processing network are corrected according to the loss value, and the training image is continuously input to repeat the above training process; if the prediction result of the image processing network meets the predetermined accuracy condition, the image processing network training is considered to be completed.
[0105] For example, after obtaining the encoding vector, the training method provided by at least one embodiment of the present disclosure further includes: updating the feature dictionary.
[0106] For example, updating the feature dictionary may include: clustering multiple pixel points according to multiple encoding vectors to obtain N second clusters, where N is a positive integer greater than or equal to K; determining the total number of pixel points included in each second cluster to obtain N groups; selecting K second clusters from the N second clusters according to the N groups, where the number of groups in the K second clusters is greater than or equal to the number of groups in NK second clusters other than the K second clusters in the N second clusters; determining the centers of the K second clusters; and using the K training feature vectors corresponding to the centers of the K second clusters as the K historical representative feature vectors for the next round of training.
[0107] The historical representative feature vector and the representative feature vector obtained in this way have the same dimension, which is convenient for calculation and does not bring additional noise to the image processing network.
[0108] For example, N=2K.
[0109] For example, when updating the feature dictionary, clustering processing of multiple encoding vectors may be re-executed, and the clustering processing method adopted may be the same as the method for obtaining the third cluster in step S50, or may be different, and the present disclosure does not impose any limitation on this.
[0110] For example, when updating the feature dictionary, the clustering result in step S50 may be reused, that is, M=N at this time, where the N second clusters are the M third clusters in step S50. At this time, there is no need to perform clustering processing again to reduce the clustering processing flow.
[0111] For example, according to multiple coding vectors, the clustering process performed on multiple pixels may include K-means clustering process or hierarchical clustering process, which is not limited in the present disclosure. The specific process of clustering process can be referred to the above content, which will not be repeated here.
[0112] For example, taking M=N=2*K as an example, that is, reusing the clustering result in step S50, the updating process of the feature dictionary is specifically described.
[0113] For example, when the training image is first input to the image processing network, K representative feature vectors Dcurr1 are obtained in step S30, and W*H pixels are divided into 2*K clusters in step S50. In essence, due to the one-to-one correspondence between the coding vector and the pixel, the W*H coding vectors are divided into 2*K clusters. At this time, the coding vector obtained in step S40 is a one-dimensional vector with a length of K.
[0114] Afterwards, K clusters with the largest number of groups are selected from the 2*K clusters, the centers of these K clusters are determined, and the training feature vectors corresponding to the centers (pixel points) of these K clusters are used as the historical feature vectors Dprev1 for the next round of training. At this time, the feature dictionary includes step S30 to obtain K representative feature vectors Dcurr1 and the training feature vectors Dprev1 corresponding to the centers of the K clusters here.
[0115] After that, the training image is input to the image processing network again, and the K representative feature vectors Dcurr2 corresponding to the currently input training image are obtained in step S30, and the W*H pixels are divided into 2*K clusters in step S50. At this time, the encoding vector obtained in step S40 is a one-dimensional vector with a length of 2*K.
[0116] Afterwards, K clusters with the largest number of groups are selected from the 2*K clusters, the centers of these K clusters are determined, and the training feature vectors corresponding to the centers (pixel points) of these K clusters are used as the historical feature vectors Dprev2 for the next round of training. At this time, the feature dictionary includes step S30 to obtain K representative feature vectors Dcurr2 and the training feature vectors Dprev2 corresponding to the centers of the K clusters here.
[0117] Repeat the above process, and continuously find the representative features that best represent the training images processed by the image processing network during the feature dictionary update process, incorporate these representative features into the feature dictionary, thereby collecting the image features of all input training images, and fusion the image features through encoding processing based on the feature dictionary. For example, image features include object information, and the object information is combined during the encoding process, so that when using the image processing network to perform image segmentation of a single image, different parts of the same object can be combined together, effectively solving the problem of parts of the same object with large appearance differences being segmented into different areas.
[0118] Moreover, in the process of updating the feature dictionary, the clustering results of the training feature vector and the clustering results of the coding vector are integrated, so that the two clustering results are mutually optimized, thereby obtaining more comprehensive image features that can better reflect the global information of the image.
[0119] For example, in other embodiments, when updating the feature dictionary, the clustering result of the training feature vector can also be used as the historical representative feature vector for the next round of training. For example, according to the training feature vector, a plurality of pixel points are clustered to obtain N clusters and the centers of the N clusters; the K clusters with the largest number of groups are selected, and the encoding vectors corresponding to the centers of the K clusters are used as the historical representative feature vector for the next round of training.
[0120] For example, as mentioned above, the training images for training an image processing network can come from all the images in a dataset.
[0121] For example, step S10 may include: obtaining a training image dataset, wherein the training image dataset includes a plurality of original images of the same type; traversing all original images in the training image dataset, and taking all original images as training images in sequence.
[0122] For example, the types of training datasets can include natural scenes, synthetic textures, pathology, etc.
[0123] For example, a large number of original images included in the data set are sequentially input into the image processing network as training images, and the image features and image information of all images in the data set are continuously collected through the feature dictionary, and the feature information in the entire data set is integrated, thereby providing more accurate guidance for the segmentation of each image and improving the image segmentation accuracy. In addition, the training method adopted by the present disclosure is unsupervised training, which greatly reduces the burden of annotation because image annotation is not required, and the image features and image information of a large number of images are continuously collected through the feature dictionary, which can not only reduce the difficulty of training and reduce the training cost, but also improve the segmentation accuracy of the model.
[0124] For example, in some embodiments, images of various sizes may be fused together as inputs to the image processing network, so that the parameter updates of the image processing network and the updates of the texture dictionary contain multi-scale information.
[0125] For example, step S10 may include: acquiring an original image; performing multi-scale processing on the original image to obtain a plurality of processed images with different sharpnesses; and performing connection processing on the plurality of processed images to obtain a training image.
[0126] For example, performing multi-scale processing on an original image to obtain a plurality of processed images with different clarity may include: obtaining a plurality of scaling ratios; performing size scaling processing on the original image according to the plurality of scaling ratios to obtain a plurality of intermediate images corresponding to the plurality of scaling ratios; performing size restoration processing on the plurality of intermediate images to obtain a plurality of processed images, wherein the sizes of the plurality of processed images are the same as the size of the original image.
[0127] For example, the formula for multi-scale processing is shown in Formula 3:
[0128]
[0129] Here, s represents the scaling factor, I s represents the processed image corresponding to the scaling ratio s, I represents the original image, resize(p1,(p2,p3)) represents the image resizing function, parameter p1 represents the image to be resized, parameter p2 represents the width of the resized image, parameter p3 represents the height of the resized image, W represents the width of the original image, and H represents the height of the original image. For example, here the units of the image width and height are both pixels.
[0130] For example, referring to Formula 3, the size of the original image I is first reduced to The intermediate image corresponding to the scaling ratio s is obtained, and then the size of the intermediate image is expanded to (W, H), that is, restored to the size of the original image I.
[0131] For example, the scaling ratio s may be 1, 2, or 4. It should be noted that, according to actual needs, other or more parameters may be used as scaling ratios to integrate multiple size information.
[0132] Through this processing method, a variety of processed images with different clarity can be obtained, and the target object in each processed image is located at the same position in the image, thus avoiding confusion for the image processing network and introducing multi-scale information for the image processing network.
[0133] For example, the training image I input It can be expressed as:
[0134] I input =concatenate({I s} s=1,2,4 ) (Formula 4)
[0135] Here, concatenate represents the concatenation processing function, and the scaling ratios are 1, 2, and 4 respectively.
[0136] For example, the original image can be a color image or a grayscale image. For example, if there are 3 scaling ratios, when the original image is a color image, the training image includes a 9-channel input image with a size of W*H, and when the original image is a color image, the training image includes a 3-channel input image with a size of W*H.
[0137] Through the above multi-scale processing, the training image integrates multi-scale information, thereby ensuring that the same object is segmented into one region in images with different sizes, rather than being segmented into different regions due to different image sizes.
[0138] For example, the original image in the data set can first perform the above-mentioned multi-scale processing, and then be input into the image processing network as a training image for training. As a result, the image processing network and the corresponding feature dictionary not only collect the image information in the data set, but also collect multiple size information.
[0139] Figure 3 A training flow chart is provided for at least one embodiment of the present disclosure.
[0140] Combine the following Figure 3 , specifically describing the execution process of the image processing network training method provided by at least one embodiment of the present disclosure.
[0141] like Figure 3 As shown, first, obtain the original image.
[0142] Afterwards, the original image is subjected to multi-scale processing to obtain three processed images. For example, processed image 1 is a processed image corresponding to a scaling ratio of 4, processed image 2 is a processed image corresponding to a scaling ratio of 2, and processed image 3 is a processed image corresponding to a scaling ratio of 1. For the specific process, refer to the relevant description of the aforementioned step S10, which will not be repeated here.
[0143] Afterwards, processed image 1, processed image 2 and processed image 3 are connected to obtain a training image and input into the image processing network.
[0144] Afterwards, the image processing network outputs W*H training feature vectors, and K representative feature vectors are obtained according to the W*H training feature vectors as part of the feature dictionary. The specific process refers to the relevant description of step S30 and will not be repeated here.
[0145] Afterwards, W*H training feature vectors are encoded based on multiple image feature vectors included in the feature dictionary to obtain W*H encoding vectors. For the specific process, refer to the relevant description of step S40, and the repeated parts will not be repeated here.
[0146] Afterwards, a plurality of pixel points are clustered according to the plurality of coding vectors to obtain pseudo segmentation labels. For the specific process, refer to the relevant description in step S50, which will not be described here. Figure 3 As shown, in the pseudo segmentation label, multiple pixels included in the training image are marked with different colors for distinction.
[0147] Afterwards, the parameters of the image processing network are updated based on the pseudo segmentation labels. For the specific process, please refer to the relevant description in step S50, which will not be repeated here.
[0148] In addition, the feature dictionary may be updated according to the encoding vector. The specific process may refer to the related content of updating the feature dictionary mentioned above, which will not be described in detail here.
[0149] After that, the above steps are repeated continuously to traverse all the original images in the data set until a trained image processing network is obtained.
[0150] Afterwards, the image processing network can be used directly for image segmentation.
[0151] Figure 4 A schematic diagram of a segmentation effect provided by an embodiment of the present disclosure.
[0152] For example, the datasets used for training include a natural scene dataset including natural scene images, a texture dataset including synthetic texture images, and a pathology dataset including pathology images.
[0153] For example, Figure 1 The unsupervised segmentation algorithm based on superpixel constraints is used as a reference solution. Figure 4 The first row of images in the figure are original images, the second row of images are annotated images after manual image annotation, the third row of images are segmentation results of the reference solution, and the fourth row of images (this solution) are segmentation results output by the image processing network obtained by the training method provided by at least one embodiment of the present disclosure. Figure 4 As shown, the segmentation result output by the image processing network trained by the training method provided by at least one embodiment of the present disclosure is better than the segmentation result of the unsupervised segmentation algorithm based on superpixel constraints.
[0154] Table 1 shows the segmentation results of the two training schemes by mean intersection over union (mIOU). In the field of computer vision deep learning image segmentation, mIOU is an important indicator to measure the accuracy of image segmentation. Generally, the larger the mIOU of the segmented image, the better the image segmentation effect.
[0155] Table 1
[0156] Natural scene dataset Texture dataset Pathology dataset Reference Solution 0.2243 0.3490 0.1375 This program 0.2399 0.4033 0.1677
[0157] It can also be seen from Table 1 that the training method provided by at least one embodiment of the present disclosure (the present solution in Table 1) obtains a better image segmentation effect of the image processing network, and the average intersection-over-union ratio in various scenarios is larger.
[0158] For example, at least one embodiment of the present disclosure also provides an image processing method. Figure 5 A schematic flowchart of an image processing method provided for at least one embodiment of the present disclosure.
[0159] like Figure 5 As shown, the image processing method includes steps S60-S70.
[0160] In step S60, an input image is acquired.
[0161] For example, the input image can be an image of the same type as the training dataset that the image processing network was trained on. For example, the input image can be a natural scene image, a synthetic texture image, or a pathological image, etc., as long as the image processing network has received training on the corresponding dataset.
[0162] In step S70, an image processing network is used to perform image segmentation processing on the input image to obtain a segmented image corresponding to the input image.
[0163] For example, the image processing network is at least partially trained according to the image processing network training method described in at least one embodiment of the present disclosure.
[0164] The training method of the image processing network has been described above and will not be repeated here.
[0165] For example, image segmentation processing includes semantic segmentation processing.
[0166] Figure 4 Table 1 shows the segmentation results of the image processing network obtained by training the training method provided by at least one embodiment of the present disclosure. Figure 4 As can be seen from Table 1, the segmentation results obtained are more accurate and have better segmentation effects.
[0167] At least one embodiment of the present disclosure further provides a training device for an image processing network. Fig. 6A A schematic block diagram of a training device provided in at least one embodiment of the present disclosure.
[0168] like Fig. 6AAs shown, the training device 100 of the image processing network may include a training image acquisition unit 101, a feature extraction unit 102, a feature dictionary acquisition unit 103, an encoding processing unit 104 and a training unit 105. These components are interconnected by a bus system and / or other forms of connection mechanisms (not shown). For example, these modules can be implemented by hardware (such as circuit) modules, software modules, or any combination of the two. The following embodiments are the same and will not be repeated here. For example, these units can be implemented by a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), a field programmable gate array (FPGA), or other forms of processing units with data processing capabilities and / or instruction execution capabilities, and corresponding computer instructions. It should be noted that Fig. 6A The components and structures of the training device 100 shown are merely exemplary and non-limiting. The training device 100 may also have other components and structures as required.
[0169] For example, the training image acquisition unit 101 is configured to acquire a training image, wherein the training image includes a plurality of pixels arranged in an array.
[0170] For example, the feature extraction unit 102 is configured to perform feature extraction processing on the training image using an image processing network to obtain a plurality of training feature vectors corresponding one-to-one to a plurality of pixel points.
[0171] For example, the feature dictionary acquisition unit 103 is configured to acquire a feature dictionary, wherein the feature dictionary includes a plurality of image feature vectors.
[0172] For example, the encoding processing unit 104 is configured to perform encoding processing on the multiple training feature vectors based on the multiple image feature vectors to obtain multiple encoding vectors corresponding to the multiple training feature vectors one by one.
[0173] For example, the training unit 105 is configured to perform image segmentation training on the image processing network according to multiple encoding vectors and multiple training feature vectors.
[0174] For example, the training unit 105 includes a training network 106, and the training network 106 includes an image processing network for obtaining a segmented image, a loss function, etc. (all not shown). The training unit 105 is used to train the image processing network to be trained to obtain a trained image processing network.
[0175] It should be noted that the training network in the training unit 105 has the same structure and function as the image processing network in the above-mentioned embodiment of the training method for the image processing network, and will not be described in detail here.
[0176] For example, the training image acquisition unit 101, the feature extraction unit 102, the feature dictionary acquisition unit 103, the encoding processing unit 104, and the training unit 105 may include codes and programs stored in the memory; the processor may execute the codes and programs to implement some or all functions of the training image acquisition unit 101, the feature extraction unit 102, the feature dictionary acquisition unit 103, the encoding processing unit 104, and the training unit 105 as described above. For example, the training image acquisition unit 101, the feature extraction unit 102, the feature dictionary acquisition unit 103, the encoding processing unit 104, and the training unit 105 may be dedicated hardware devices to implement some or all functions of the training image acquisition unit 101, the feature extraction unit 102, the feature dictionary acquisition unit 103, the encoding processing unit 104, and the training unit 105 as described above. For example, the training image acquisition unit 101, the feature extraction unit 102, the feature dictionary acquisition unit 103, the encoding processing unit 104, and the training unit 105 may be a circuit board or a combination of multiple circuit boards to implement the functions as described above. In an embodiment of the present application, the circuit board or the combination of multiple circuit boards may include: (1) one or more processors; (2) one or more non-temporary memories connected to the processors; and (3) firmware stored in the memories that is executable by the processors.
[0177] It should be noted that the training image acquisition unit 101 can be used to implement Figure 2 In step S10 shown in FIG. 1 , the feature extraction unit 102 may be used to implement Figure 2 In step S20 shown in FIG. 1 , the feature dictionary acquisition unit 103 may be used to implement Figure 2 In step S30 shown in FIG. 1 , the encoding processing unit 104 may be used to implement Figure 2 In step S40 shown in FIG. 1 , the training unit 105 may be used to implement Figure 2 Step S50 is shown. Therefore, for the specific description of the functions that can be implemented by the training image acquisition unit 101, the feature extraction unit 102, the feature dictionary acquisition unit 103, the encoding processing unit 104 and the training unit 105, reference can be made to the relevant description of steps S10 to S50 in the above-mentioned embodiment of the image processing method, and the repeated parts will not be repeated. In addition, the image processing device 100 can achieve technical effects similar to those of the above-mentioned image processing method, which will not be repeated here.
[0178] It should be noted that in the embodiments of the present disclosure, the image processing device 100 may include more or fewer circuits or units, and the connection relationship between the various circuits or units is not limited and can be determined according to actual needs. The specific configuration of each circuit or unit is not limited and can be composed of analog devices according to circuit principles, or can be composed of digital chips, or in other applicable ways.
[0179] At least one embodiment of the present disclosure further provides an image processing device, Figure 6B A schematic block diagram of an image processing device provided for at least one embodiment of the present disclosure.
[0180] like Figure 6B As shown, the image processing device 200 may include an image input unit 201 and a segmentation processing unit 202. These components are interconnected via a bus system and / or other forms of connection mechanisms (not shown). Figure 6B The components and structures of the image processing device 200 shown are merely exemplary and not restrictive. The image processing device 200 may also have other components and structures as required.
[0181] For example, the input image acquisition unit 201 is configured to acquire an input image.
[0182] For example, the processing unit 202 is configured to perform image segmentation processing on the input image using an image processing network to obtain a segmented image corresponding to the input image.
[0183] For example, the image processing network is at least partially trained according to the training method described in any embodiment of the present disclosure. The specific process of the training can refer to the relevant description in the embodiment of the training method of the above-mentioned image processing network, and the repeated parts will not be repeated.
[0184] At least one embodiment of the present disclosure further provides an electronic device, Figure 7 A schematic diagram of an electronic device provided according to at least one embodiment of the present disclosure.
[0185] For example, Figure 7 As shown, the electronic device includes a processor 701, a communication interface 702, a memory 703 and a communication bus 704. The processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704. The processor 701, the communication interface 702, the memory 703 and other components can also communicate through a network connection. The present disclosure does not limit the type and function of the network. It should be noted that Figure 7 The components of the electronic device shown are merely exemplary and non-limiting. The electronic device may also have other components according to actual application requirements.
[0186] For example, the memory 703 is used to store computer-readable instructions non-transiently. When the processor 701 is used to execute the computer-readable instructions, the training method of the image processing network or the image processing method according to any of the above embodiments is implemented. The specific implementation of each step of the training method of the image processing network or the image processing method and the related explanation content can be referred to the embodiments of the training method of the image processing network or the image processing method, which will not be repeated here.
[0187] For example, the training method of the image processing network or other implementation methods of the image processing method implemented by the processor 701 executing the computer-readable instructions stored in the memory 703 are the same as the implementation methods mentioned in the aforementioned method embodiment part, and will not be repeated here.
[0188] For example, the communication bus 704 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industrial Standard Architecture (EISA) bus, etc. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
[0189] For example, the communication interface 702 is used to implement communication between the electronic device and other devices.
[0190] For example, the processor 701 and the memory 703 may be arranged on a server side (or a cloud side).
[0191] For example, the processor 701 can control other components in the electronic device to perform the desired functions. The processor 701 can be a device with data processing capabilities and / or program execution capabilities, such as a central processing unit (CPU), a network processor (NP), a tensor processing unit (TPU), or a graphics processing unit (GPU); it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The central processing unit (CPU) can be an X86 or ARM architecture, etc.
[0192] For example, the memory 703 may include any combination of one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disk read-only memory (CD-ROM), USB memory, flash memory, etc. One or more computer-readable instructions may be stored on the computer-readable storage medium, and the processor 701 may execute the computer-readable instructions to implement various functions of the electronic device. Various applications and various data may also be stored in the storage medium.
[0193] For example, in some embodiments, the electronic device may further include an image acquisition component. The image acquisition component is used to acquire images. The memory 703 is also used to store the acquired images.
[0194] For example, the image acquisition component may be a camera of a smartphone, a camera of a tablet computer, a camera of a personal computer, a lens of a digital camera, or even a webcam.
[0195] For example, for a detailed description of the process of an electronic device executing a training method for an image processing network or an image processing method, reference may be made to the relevant description in the embodiments of the training method for an image processing network or an image processing method, and repeated descriptions will not be repeated.
[0196] Figure 8 A schematic diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure. Figure 8 As shown, the storage medium 800 may be a non-transitory computer-readable storage medium, and one or more computer-readable instructions 801 may be non-transitory stored on the storage medium 800. For example, when the computer-readable instructions 801 are executed by a processor, one or more steps in the image processing method or the image processing network training method described above may be executed.
[0197] For example, the storage medium 800 may be applied to the above-mentioned electronic device. For example, the storage medium 800 may include a memory in the electronic device.
[0198] For example, the storage medium may include a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), flash memory, or any combination of the above storage media, or other applicable storage media.
[0199] For example, the description of the storage medium 800 may refer to the description of the memory in the embodiment of the electronic device, and the repeated parts will not be repeated.
[0200] Fig. 9 A schematic diagram of a hardware environment provided by at least one embodiment of the present disclosure. The electronic device provided by the present disclosure can be applied in an Internet system.
[0201] use Fig. 9 The computer system provided in can implement the functions of the image processing device and / or electronic device involved in the present disclosure. Such computer systems may include personal computers, laptops, tablet computers, mobile phones, personal digital assistants, smart glasses, smart watches, smart rings, smart helmets and any smart portable devices or wearable devices. The specific system in this embodiment uses a functional block diagram to explain a hardware platform including a user interface. This computer device can be a general-purpose computer device or a computer device with a specific purpose. Both computer devices can be used to implement the image processing device and / or electronic device in this embodiment. The computer system may include any components that implement the information required to implement the image processing currently described. For example, the computer system can be implemented by a computer device through its hardware devices, software programs, firmware and their combinations. For convenience, Fig. 9 Only one computer device is depicted in the figure, but the relevant computer functions of the information required for realizing image processing described in this embodiment can be implemented in a distributed manner by a group of similar platforms to disperse the processing load of the computer system.
[0202] like Fig. 9As shown, the computer system may include a communication port 250, connected to which is a network for realizing data communication. For example, the computer system can send and receive information and data through the communication port 250, that is, the communication port 250 can realize wireless or wired communication between the computer system and other electronic devices to exchange data. The computer system may also include a processor group 220 (i.e., the processor described above) for executing program instructions. The processor group 220 may be composed of at least one processor (e.g., a CPU). The computer system may include an internal communication bus 210. The computer system may include different forms of program storage units and data storage units (i.e., the memory or storage medium described above), such as a hard disk 270, a read-only memory (ROM) 230, and a random access memory (RAM) 240, which can be used to store various data files used for computer processing and / or communication, and possible program instructions executed by the processor group 220. The computer system may also include an input / output component 260, which is used to realize input / output data flow between the computer system and other components (e.g., a user interface 280, etc.).
[0203] Typically, the following devices can be connected to the input / output component 260: input devices such as a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices such as a display (e.g., an LCD, an OLED display, etc.), a speaker, a vibrator, etc.; storage devices including, for example, a magnetic tape, a hard disk, etc.; and a communication interface.
[0204] Although Fig. 9 A computer system having various devices is shown, but it should be understood that the computer system is not required to have all of the devices shown, and the computer system may alternatively have more or fewer devices.
[0205] There are a few points to note about this disclosure:
[0206] (1) The drawings of the embodiments of the present disclosure only relate to the structures related to the embodiments of the present disclosure, and other structures may refer to the general design.
[0207] (2) For the sake of clarity, the thickness and size of layers or structures are exaggerated in the drawings used to describe the embodiments of the present invention. It is understood that when an element such as a layer, film, region, or substrate is referred to as being "on" or "under" another element, the element may be "directly" "on" or "under" the other element, or there may be intervening elements.
[0208] (3) In the absence of conflict, the embodiments of the present disclosure and the features therein may be combined with each other to obtain new embodiments.
[0209] The above description is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. The protection scope of the present disclosure shall be based on the protection scope of the claims.
Claims
1. A training method for an image processing network, comprising: Acquire a training image, wherein the training image includes a plurality of pixel points arranged in an array; Using the image processing network to perform feature extraction processing on the training image to obtain a plurality of training feature vectors corresponding to the plurality of pixel points one by one; Acquire a feature dictionary, wherein the feature dictionary includes a plurality of image feature vectors; Performing encoding processing on the multiple training feature vectors based on the multiple image feature vectors to obtain multiple encoding vectors corresponding one-to-one to the multiple training feature vectors; Performing image segmentation training on the image processing network according to the multiple encoding vectors and the multiple training feature vectors, Among them, obtaining the feature dictionary includes: According to the multiple training feature vectors, clustering processing is performed on the multiple pixel points to obtain K representative feature vectors, wherein the multiple image feature vectors include the K representative feature vectors and K historical representative feature vectors, and K is a positive integer; The encoding process of the plurality of training feature vectors based on the plurality of image feature vectors to obtain a plurality of encoding vectors corresponding one-to-one to the plurality of training feature vectors includes: For the i-th training feature vector among the multiple training feature vectors: Performing similarity calculations on the i-th training feature vector and the multiple image feature vectors respectively to obtain multiple similarity values corresponding to the i-th training feature vector; The multiple similarity values are used as the encoding vector corresponding to the i-th training feature vector, Wherein, i is a positive integer and is less than or equal to the total number of the plurality of training feature vectors; The image processing network is trained for image segmentation according to the multiple encoding vectors and the multiple training feature vectors, including: Performing clustering processing on the multiple pixel points according to the multiple encoding vectors to obtain pseudo segmentation labels corresponding to the training image; The image processing network is trained according to the pseudo segmentation labels and the multiple training feature vectors.
2. The training method according to claim 1, wherein: The training image does not have a corresponding segmentation label, and the training performed on the image processing network is unsupervised training.
3. The training method according to claim 1, wherein: According to the multiple training feature vectors, clustering is performed on the multiple pixel points to obtain K representative feature vectors, including: According to the multiple training feature vectors, performing K-means clustering processing or hierarchical clustering processing on the multiple pixel points to divide the multiple pixel points into K first clusters; Determine the centers of the K first clusters; The K training feature vectors corresponding to the centers of the K first clusters are used as the K representative feature vectors.
4. The training method according to claim 1, wherein: After obtaining a plurality of encoding vectors corresponding one to one to the plurality of training feature vectors, the method further comprises: The feature dictionary is updated.
5. The training method according to claim 4, wherein: Updating the feature dictionary includes: Clustering the plurality of pixels according to the plurality of encoding vectors to obtain N second clusters, where N is a positive integer greater than or equal to K; Determine the total number of pixels included in each second cluster to obtain N groups; According to the N numbers of groups, K second clusters are selected from the N second clusters, wherein the numbers of groups in the K second clusters are all greater than or equal to the numbers of groups in NK second clusters other than the K second clusters in the N second clusters; Determining the centers of the K second clusters; The K training feature vectors corresponding to the centers of the K second clusters are used as K historical representative feature vectors for the next round of training.
6. The training method according to claim 5, wherein: N=2K.
7. The training method according to claim 1, wherein: Clustering the plurality of pixels according to the plurality of encoding vectors to obtain pseudo segmentation labels corresponding to the training image includes: Performing clustering processing on the plurality of pixel points according to the plurality of encoding vectors to obtain M third clusters, where M is a positive integer; The plurality of pixels are labeled according to the M third clusters to obtain the pseudo segmentation labels, wherein the pixels included in each third cluster are labeled as belonging to the same category.
8. The training method according to claim 1, wherein: Training the image processing network according to the pseudo segmentation label and the plurality of training feature vectors includes: Calculating a loss value of the image processing network by using a loss function corresponding to the image processing network according to the pseudo segmentation label and the plurality of training feature vectors; and Modifying parameters of the image processing network based on the loss value; When the loss value corresponding to the image processing network does not meet a predetermined condition, continue to input the training image to repeat the training process.
9. The training method according to any one of claims 1 to 8, wherein: Get training images, including: Get the original image; Performing multi-scale processing on the original image to obtain a plurality of processed images with different sharpnesses; The plurality of processed images are connected to obtain the training image.
10. The training method according to claim 9, wherein: The original image is subjected to multi-scale processing to obtain a plurality of processed images with different sharpnesses, including: Get multiple zoom ratios; Performing size scaling processing on the original image according to the multiple scaling ratios to obtain multiple intermediate images corresponding to the multiple scaling ratios respectively; The multiple intermediate images are subjected to size restoration processing to obtain the multiple processed images, wherein the sizes of the multiple processed images are the same as the size of the original images.
11. The training method according to any one of claims 1 to 8, wherein: Get training images, including: Acquire a training image data set, wherein the training image data set includes a plurality of original images of the same type; All original images in the training image data set are traversed, and all the original images are used as the training images in sequence.
12. An image processing method, comprising: Get the input image; Using an image processing network to perform image segmentation processing on the input image to obtain a segmented image corresponding to the input image, Wherein, the image processing network is at least partially trained according to the training method according to any one of claims 1-11.
13. The image processing method according to claim 12, wherein: The image segmentation processing includes semantic segmentation processing.
14. A training device for an image processing network, comprising: A training image acquisition unit, configured to acquire a training image, wherein the training image includes a plurality of pixels arranged in an array; a feature extraction unit configured to perform feature extraction processing on the training image using the image processing network to obtain a plurality of training feature vectors corresponding one-to-one to the plurality of pixel points; A feature dictionary acquisition unit, configured to acquire a feature dictionary, wherein the feature dictionary includes a plurality of image feature vectors; An encoding processing unit configured to perform encoding processing on the plurality of training feature vectors based on the plurality of image feature vectors to obtain a plurality of encoding vectors corresponding one-to-one to the plurality of training feature vectors; a training unit configured to perform image segmentation training on the image processing network according to the multiple encoding vectors and the multiple training feature vectors, Among them, obtaining the feature dictionary includes: According to the multiple training feature vectors, clustering processing is performed on the multiple pixel points to obtain K representative feature vectors, wherein the multiple image feature vectors include the K representative feature vectors and K historical representative feature vectors, and K is a positive integer; The encoding process of the plurality of training feature vectors based on the plurality of image feature vectors to obtain a plurality of encoding vectors corresponding one-to-one to the plurality of training feature vectors includes: For the i-th training feature vector among the multiple training feature vectors: Performing similarity calculations on the i-th training feature vector and the multiple image feature vectors respectively to obtain multiple similarity values corresponding to the i-th training feature vector; The multiple similarity values are used as the encoding vector corresponding to the i-th training feature vector, Wherein, i is a positive integer and is less than or equal to the total number of the plurality of training feature vectors; The image processing network is trained for image segmentation according to the multiple encoding vectors and the multiple training feature vectors, including: Performing clustering processing on the multiple pixel points according to the multiple encoding vectors to obtain pseudo segmentation labels corresponding to the training image; The image processing network is trained according to the pseudo segmentation labels and the multiple training feature vectors.
15. An image processing device, comprising: An input image acquisition unit, configured to acquire an input image; a processing unit configured to perform image segmentation processing on the input image using an image processing network to obtain a segmented image corresponding to the input image, Wherein, the image processing network is at least partially trained according to the training method according to any one of claims 1-11.
16. An electronic device comprising: A memory non-transitorily stores computer executable instructions; a processor configured to execute the computer executable instructions, Wherein, when the computer executable instructions are executed by the processor, the training method of the image processing network according to any one of claims 1-11 or the image processing method according to any one of claims 12-13 is implemented.
17. A non-transitory computer-readable storage medium, wherein: The non-transitory computer-readable storage medium stores computer-executable instructions, When the computer executable instructions are executed by a processor, the training method of the image processing network according to any one of claims 1-11 or the image processing method according to any one of claims 12-13 is implemented.
