Mask image modeling self-supervised learning method and device based on unified information flow

Abstract

The invention provides a mask image modeling self-supervised learning method and device based on a unified information flow. The method comprises the following steps: carrying out image partitioning on a to-be-processed image of a self-supervised learning task to obtain equally divided pixel blocks; calling a feature encoder to perform decoupling processing based on information flow on the equally divided pixel blocks to obtain deep representation information; based on a fractal space filling curve normal form, performing fractal sequence construction on the deep representation information to obtain fractal sequence representation information, and calling a feature decoder to decode to obtain a feature predicted value; and performing linear mapping processing on the feature prediction value to obtain a target prediction result of the self-supervised learning task, and constructing a self-supervised learning loss value for training the self-supervised learning model. Through the method and the device, the problems that the image representation learning ability is reduced and the universality is poor during mask image modeling due to the fact that the dependency relationship between image block features and image modal two-dimensional features are not concerned during image feature construction in the prior art are solved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a mask image modeling self-supervised learning method and device based on unified information flow. Background Art

[0002] As a generative visual self-supervised learning task, mask image modeling has made significant academic progress and has been widely used due to its powerful representation learning ability. It usually relies on autoregressive models or autoencoders to reconstruct damaged images without any label information to complete the pre-training task.

[0003] In the prior art, the mask image modeling often uses autoencoders and autoregressive models to model the data. The autoencoder usually treats all masked image blocks equally and completes the prediction at the same time step. It adopts an architecture with both bidirectional and symmetric information flow. Although the prediction is relatively simple, the prediction process does not pay attention to the dependencies between the image block features. Although the autoregressive model takes into account the sequentiality between images, it serializes the image features to characterize the dependencies between the image block features. However, for the image mode, the image blocks are still scanned in the same sequence scanning method as natural language (from left to right and then from top to bottom) when the features are serialized. Although a random serialization method was subsequently proposed for sequence scanning, both the sequence scanning method like natural language and the random serialization method lack consideration of the two-dimensional characteristics of the image modality, resulting in a decrease in the image representation learning ability in mask image modeling, and are limited to certain target task scenarios, with poor versatility. Summary of the invention

[0004] The present invention provides a self-supervised learning method and device for mask image modeling based on unified information flow, which is used to solve the technical problems that in the prior art, when constructing serialized features of mask image modeling, no attention is paid to the dependencies between image block features, and the two-dimensional characteristics of the image modality are not considered. The method of natural language serial scanning or random serialization is still used, which leads to a decrease in the image representation learning ability in mask image modeling and poor versatility.

[0005] The present invention provides a self-supervised learning method for mask image modeling based on unified information flow, which is applied to a self-supervised learning model. The self-supervised learning model includes an image block encoder, a feature encoder, and a feature decoder. The method includes the following steps: Acquire an image to be processed for a self-supervised learning task, and call the image block encoder to perform image block processing on the image to be processed to obtain a plurality of equally divided pixel blocks; Calling the feature encoder to perform information flow-based decoupling processing on the equally divided pixel blocks to obtain deep representation information of the image to be processed; Based on the fractal space filling curve paradigm, a fractal sequence is constructed for the deep representation information to obtain fractal sequence representation information, and the feature decoder is called to perform feature decoding processing on the fractal sequence representation information to obtain a feature prediction value for the self-supervised learning task; The feature prediction value is linearly mapped to obtain a target prediction result of the self-supervised learning task, and a self-supervised learning loss value is constructed according to the target prediction result, wherein the self-supervised learning loss value is used to train the self-supervised learning model.

[0006] In some embodiments, calling the image block encoder to perform image block processing on the image to be processed to obtain a plurality of equally divided pixel blocks includes: Performing embedding processing on the image to be processed to obtain a marked embedding sequence; A binary mask sequence corresponding to the image to be processed is obtained, and the label embedding sequence is subjected to image masking by using the binary mask sequence to obtain visible pixel blocks and mask pixel blocks as a plurality of equally divided pixel blocks of the image to be processed.

[0007] In some embodiments, the information stream includes a proprioceptive information stream and an exo-sense information stream; The visible pixel blocks in the equally divided pixel blocks have a combination feature, the combination feature includes an image pixel feature and a pixel position feature, and the mask pixel blocks in the equally divided pixel blocks have a mask position feature; The proprioceptive perception information stream includes a first mapping relationship between combination features or between mask position features; The external body perception information flow includes a second mapping relationship between the combined feature and the mask position feature, or between the mask position feature and the combined feature; In some embodiments, calling the feature encoder to perform information flow-based decoupling processing on the equally divided pixel blocks to obtain deep representation information of the image to be processed includes: When the equally divided pixel blocks have a first mapping relationship, calling the attention network of the feature encoder to perform a self-attention mechanism on the equally divided pixel blocks to obtain attention output information; When the equally divided pixel blocks have a second mapping relationship, calling the attention network of the feature encoder to perform a cross attention mechanism on the equally divided pixel blocks to obtain attention output information; The attention output information is linearly mapped through a feedforward network in the feature encoder to obtain deep representation information of the image to be processed.

[0008] In some embodiments, the fractal space filling curve paradigm includes a Hilbert curve paradigm or a Z-order curve paradigm, and the fractal sequence is constructed for the deep representation information based on the fractal space filling curve paradigm to obtain the fractal sequence representation information, including: The deep representation information is subjected to fractal sequence scanning through the Hilbert curve paradigm, and the characteristic fractal sequences obtained by the sequence scanning are spliced ​​to obtain fractal sequence representation information; or the deep representation information is subjected to fractal sequence scanning through the Z-order curve paradigm, and the characteristic fractal sequences obtained by the sequence scanning are spliced ​​to obtain fractal sequence representation information.

[0009] In some embodiments, the image to be processed has a real label, and the real label is constructed based on the HOG feature map generated by the image to be processed, and the self-supervised learning loss value is constructed based on the target prediction result, including: Determine the difference between the true label and the target prediction result; An L2 norm loss function is constructed according to the difference, and the L2 norm loss function is used as a self-supervised learning loss value.

[0010] The present invention also provides a mask image modeling self-supervised learning device based on unified information flow, which includes the following modules: An acquisition module is used to acquire an image to be processed for a self-supervised learning task, and call the image block encoder to perform image block processing on the image to be processed to obtain a plurality of equally divided pixel blocks; A decoupling module, used for calling the feature encoder to perform information flow-based decoupling processing on the equally divided pixel blocks to obtain deep representation information of the image to be processed; A construction module is used to construct a fractal sequence for the deep representation information based on a fractal space filling curve paradigm to obtain fractal sequence representation information, and call the feature decoder to perform feature decoding processing on the fractal sequence representation information to obtain a feature prediction value for the self-supervised learning task; A training module is used to perform linear mapping processing on the feature prediction value to obtain the target prediction result of the self-supervised learning task, and construct a self-supervised learning loss value based on the target prediction result, wherein the self-supervised learning loss value is used to train the self-supervised learning model.

[0011] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the self-supervised learning method for mask image modeling based on unified information flow as described in any one of the above-mentioned methods.

[0012] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described mask image modeling self-supervised learning methods based on unified information flow.

[0013] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements any of the above-described mask image modeling self-supervised learning methods based on unified information flow.

[0014] The mask image modeling self-supervised learning method and device based on unified information flow provided by the present invention divides the image to be processed for the self-supervised learning task into multiple equally divided pixel blocks, and then calls the feature encoder to perform information flow-based decoupling processing on the equally divided pixel blocks to obtain deep representation information. Thus, image features are extracted on the basis of decoupling the information flow between equally divided pixel blocks. Compared with the prior art that does not pay attention to the dependency between image blocks and directly extracts image features at the same time step, image representation can be extracted more accurately. Next, the fractal space filling curve paradigm is used to construct a fractal sequence for the extracted deep representation information to obtain fractal sequence representation information for feature decoding to obtain feature prediction values. Therefore, before feature decoding, a fractal space filling curve paradigm is designed to construct fractal serialization for representation information with different information flows. Compared with the prior art that uses a natural language scanning sequence or random serialization to construct a feature sequence, the two-dimensional characteristics between image blocks can be fully considered, and the image representation accuracy can be further improved. Moreover, the fractal space filling curve paradigm can be applied to various two-dimensional image scenes and has a certain versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced one by one below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 It is a flow chart of the self-supervised learning method for mask image modeling based on unified information flow provided by the present invention.

[0017] Figure 2 It is a schematic diagram of attention calculation based on information flow provided by the present invention.

[0018] Figure 3 It is a schematic diagram of the autoencoding paradigm provided in the prior art.

[0019] Figure 4 It is a schematic diagram of the autoregressive paradigm provided in the prior art.

[0020] Figure 5 It is a schematic diagram of sequence processing of multiple autoregressive paradigms provided by the present invention.

[0021] Figure 6 It is a schematic diagram of attention visualization analysis on images of different test data sets provided by the present invention.

[0022] Figure 7 It is a structural schematic diagram of the mask image modeling self-supervised learning device based on unified information flow provided by the present invention.

[0023] Figure 8 It is a schematic diagram of the physical structure of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] Combine the following Figure 1 Describe the self-supervised learning method of mask image modeling based on unified information flow of the present invention, Figure 1 is a flow chart of the self-supervised learning method for mask image modeling based on unified information flow provided by the present invention, such as Figure 1 As shown, the method includes the following steps.

[0026] Step 101: obtain an image to be processed for a self-supervised learning task, and call an image block encoder to perform image block processing on the image to be processed to obtain a plurality of equally divided pixel blocks.

[0027] First, the mask image modeling self-supervised learning method based on unified information flow provided by the present invention is applied to a self-supervised learning model. The self-supervised learning model is generally trained through self-supervised learning tasks, and the self-supervised learning tasks can be image target detection (recognition), image semantic segmentation, image (feature) description, etc. The architecture of the self-supervised learning model specifically includes an image block encoder, a feature encoder, and a feature decoder.

[0028] First, according to the self-supervised learning task, the image to be processed for the self-supervised learning task is obtained as the input image of the self-supervised learning model, denoted as , , H, W, C represent the height, width and channel of the input image respectively, and then call the image block encoder in the self-supervised learning model to divide the image to be processed into blocks to obtain multiple equally divided pixel blocks. Here, the image block encoder is a convolutional neural network, and the convolution kernel size of the convolution layer is configured to be 16 and the step size is 16. Input into the convolutional layer of the convolutional neural network, embed the image to be processed, and obtain the labeled embedding sequence, recorded as , , P is the side length of the image block, N represents the number of equally divided pixel blocks, where, .

[0029] Next, we obtain the binary mask sequence corresponding to the image to be processed, denoted as , embedding sequence of labels by binary mask sequence Perform image masking to obtain visible pixel blocks And the mask pixel block , as multiple equally divided pixel blocks of the image to be processed. This equally divided pixel block is the feature map of the image to be processed, denoted as , and in the process of image masking, a special placeholder [MASK] can also be determined, recorded as .

[0030] Step 102: Call a feature encoder to perform information flow-based decoupling processing on equally divided pixel blocks to obtain deep representation information of the image to be processed.

[0031] After determining a plurality of equally divided pixel blocks, next, a feature encoder is called to perform information flow-based decoupling processing on the equally divided pixel blocks to obtain deep representation information of the image to be processed.

[0032] Before extracting deep representation information, it is necessary to consider that starting from the mask operation, each equally divided pixel block has two kinds of meta-information: pixel and position. Therefore, when the equally divided pixel block is a visible pixel block (unmasked pixel block), the visible pixel block has a combination feature, and the combination feature includes image pixel features and pixel position features, which means that the visible pixel block has two kinds of meta-information. Correspondingly, when the equally divided pixel block is a masked pixel block (masked pixel block), the masked pixel block only has a mask position feature, which means that the masked pixel block only has one kind of meta-information.

[0033] Based on this, in an embodiment of the present invention, the feature dependency relationship between equally divided pixel blocks is decoupled into four information flows, and these four information flows are divided into body perception information flow and external body perception information flow. Among them, the body perception information flow is divided into two types, namely the first mapping relationship between the combined features of the visible pixel blocks, or the first mapping relationship between the mask position features of the mask pixel blocks. This first mapping relationship characterizes the information flow between two adjacent visible pixel blocks or two adjacent mask pixel blocks. It can be seen that the information flow between the visible pixel blocks and the visible pixel blocks is a bidirectional semantic and position information fusion information flow, while the information flow between the mask pixel blocks and the mask pixel blocks is only a bidirectional position information fusion information flow.

[0034] The external body perception information flow is also divided into two types, namely the second mapping relationship between the combined features of the visible pixel block and the mask position features of the mask pixel block, or the second mapping relationship between the mask position features of the mask pixel block and the combined features of the visible pixel block. This second mapping relationship represents the information flow between the visible pixel block and the adjacent mask pixel block, or between the mask pixel block and the adjacent visible pixel block. It can be seen that the information flow from the visible pixel block to the mask pixel block is a one-way semantic and position information propagation information flow, and the information flow from the mask pixel block to the visible pixel block is a one-way position information propagation information flow.

[0035] According to the decoupled information flow, the feature encoder in the self-supervised learning model is called to extract deep representation information. The feature encoder (for example, a Transformer) includes an attention network and a feedforward network (for example, a multi-layer perceptron). The attention network includes 12 decoupled fractal attention modules, and the feedforward network includes 12 forward feedback modules. The above 24 modules are alternately stacked to form a feature encoder. The present invention adopts a dual-stream modeling method based on context-query to equally divide pixel blocks. The feature extraction is described in detail below.

[0036] When the equally divided pixel blocks have the first mapping relationship, the attention network of the feature encoder is called to perform a self-attention mechanism on the equally divided pixel blocks to obtain attention output information.

[0037] Here, when the equally divided pixel blocks have the first mapping relationship, it means that the feature dependency relationship between the equally divided pixel blocks is decoupled from the ontological perception information flow (i.e., the information flow of bidirectional semantics and position information fusion between visible pixel blocks and visible pixel blocks, and the information flow of bidirectional position information fusion between mask pixel blocks and mask pixel blocks), and the self-attention calculation is performed on the equally divided pixel blocks through the attention network of the feature encoder. Figure 2 , Figure 2 Schematic diagram of attention calculation based on information flow provided by the present invention. Based on the information flow of bidirectional semantic and position information fusion between visible pixel blocks and visible pixel blocks, the self-attention calculation is as follows Figure 2 As shown in (1), based on the image pixel features and the [MASK] mask token (i.e., the special placeholder of [MASK] ) Calculate the query vector , key vector Sum value vector , and then perform self-attention calculation to obtain the attention output information.

[0038] Based on the information flow between masked pixel blocks and masked pixel blocks, which is only for bidirectional position information fusion, the self-attention calculation is as follows: Figure 2 As shown in (2), the query vector is calculated based on the position features of the equally divided pixel blocks (including pixel position features and mask position features) , key vector Sum value vector , and then perform self-attention calculation to obtain the attention output information.

[0039] In the process of self-attention calculation, the input visible pixel block is transformed into and mask pixel blocks Linear projection into three high-dimensional vector subspaces to obtain the corresponding query vector ( ), key vector ( ) and the value vector ( ), and then perform attention calculation to obtain the attention output information, recorded as The attention calculation process is as follows: (1)

[0040] In the above formula (1), represents the eigenvector multiplication calculation, T represents the matrix transpose, represents the characteristic modulation coefficient, K represents the feature vector dimension, and i represents the i-th equally divided pixel block.

[0041] When the equally divided pixel blocks have the second mapping relationship, the attention network of the feature encoder is called to perform a cross-attention mechanism on the equally divided pixel blocks to obtain attention output information. Here, when the equally divided pixel blocks have the second mapping relationship, it means that the feature dependency relationship between the equally divided pixel blocks is decoupled from the external body perception information flow (that is, the information flow of unidirectional semantic and position information propagation between the visible pixel blocks and the mask pixel blocks, and the information flow of unidirectional position information propagation between the mask pixel blocks and the visible pixel blocks). Then, the attention network of the feature encoder is used to perform cross-attention calculation on the equally divided pixel blocks. Based on the information flow of unidirectional semantic and position information propagation between the visible pixel blocks and the mask pixel blocks, the cross-attention calculation is as follows: Figure 2As shown in (3), in the cross process, the query vector is calculated based on the image pixel features and the [MASK] mask token , calculate the key vector based on the position features of the equally divided pixel blocks (including pixel position features and mask position features) Sum value vector , and then perform attention calculation to obtain attention output information. The inference calculation of the information flow of one-way position information propagation from the combined feature to the mask position feature is realized.

[0042] Based on the information flow of unidirectional position information propagation from masked pixel blocks to visible pixel blocks, the cross attention calculation is as follows: Figure 2 As shown in (4), during the crossover process, the query vector is calculated based on the position features of the equally divided pixel blocks (including pixel position features and mask position features). , calculate the key vector based on the image pixel features and the [MASK] mask token Sum value vector , and then perform attention calculation to obtain attention output information. The inference calculation of the information flow of unidirectional position information propagation from mask position features to combined features is realized.

[0043] The attention output information corresponding to various information flows is obtained through attention network calculation Afterwards, the attention output information is fed back through the feedforward network in the feature encoder Perform linear mapping to obtain the deep representation information of the image to be processed, which is recorded as .

[0044] The embodiment of the present invention divides the image to be processed into multiple equally divided pixel blocks, and then calls a feature encoder to perform information flow-based decoupling processing on the equally divided pixel blocks to obtain deep representation information. Thus, image features are extracted on the basis of decoupling the information flow between the equally divided pixel blocks. Compared with the prior art that does not consider the dependency between image blocks and directly extracts image features at the same time step, image representation can be extracted more accurately.

[0045] Continue to see Figure 1 , step 103, based on the fractal space filling curve paradigm, the deep representation information is constructed into a fractal sequence to obtain the fractal sequence representation information, and the feature decoding encoder is called to perform feature decoding processing on the fractal sequence representation information to obtain the feature prediction value of the self-supervised learning task.

[0046] After extracting features in step 102, the extracted features need to be decoded according to the learning objectives of self-supervised learning and mapped into feature prediction values ​​for the learning objectives to perform the self-supervised learning task. However, before feature decoding, the extracted deep representation information needs to be Serialization construction is generally completed through the serialization paradigm corresponding to the information flow, which includes the autoencoding paradigm and the autoregressive paradigm. Figure 3 As shown, Figure 3 is a schematic diagram of the autoencoder paradigm provided in the prior art. The autoencoder paradigm usually adopts a bidirectional and symmetric architecture ([M] belongs to the mask token) according to the image pixel feature x ( ) and image position feature p( ) to extract deep representation information z, where , but does not consider the dependencies between features. Figure 4 As shown, Figure 4 It is a schematic diagram of the autoregressive paradigm provided in the prior art. Although the autoregressive paradigm focuses on the sequentiality between image pixel features, that is, the available contextual information should change dynamically with its position in the sequence. Figure 4 In the image, pixel features Start by extracting context information T times until deep representation information is obtained As the final context information, the process is equivalent to processing a sequence of T elements in sequence , the context information obtained each time is different according to the different positions of the elements in the sequence. Since there is a connection between each adjacent element in the sequence, feature serialization can theoretically express the dependency between image pixel features. However, the autoregressive paradigm does not consider the two-dimensional characteristics of the image modality (the element sequence is only one-dimensional), because there may also be a dependency between two non-adjacent image pixel features.

[0047] The autoregressive paradigm often requires the input image feature information to be serialized first, and serialization processing generally involves scanning the feature information. Figure 5 , Figure 5 Schematic diagram of sequence processing of multiple autoregressive paradigms provided by the present invention, showing the process of processing four feature sequences for 64 equally divided pixel blocks (respectively marked as 0-63), wherein Figure 5 (a) shows the traditional autoregressive paradigm, which is a natural language-based sequence scanning method that scans in a strict line-by-line manner, and the sequence representation information obtained by the final scan is fixed. Figure 5 (b) represents the scanning method based on the random serialization paradigm. The direction of each sequence scan is random, so the sequence representation information obtained by each scan is not fixed. Both methods cannot accurately express the dependency between equally divided pixel blocks.

[0048] Based on this, the embodiment of the present invention designs a fractal space filling curve paradigm, and based on the fractal space filling curve paradigm, a fractal sequence is constructed for the deep representation information to obtain fractal sequence representation information. Among them, the fractal space filling curve paradigm includes the Hilbert curve paradigm or the Z-order curve paradigm, which is essentially also a fractal autoregressive paradigm.

[0049] The Hilbert curve paradigm is designed with the help of the Hilbert curve definition in fractal geometry. When constructing a fractal sequence, the Hilbert curve paradigm is used to perform a fractal sequence scan on the deep representation information, and the characteristic fractal sequence obtained by the sequence scan is spliced ​​to obtain the fractal sequence representation information. The schematic diagram of the analysis sequence scan is shown in Figure 5 As shown in (c) in the figure, the extracted fractal sequence representation information can accurately express the dependency relationship between each equally divided pixel block without causing the loss of feature information.

[0050] Similarly, the Z-order curve paradigm is designed with the help of the Z-order curve definition commonly used in the spatial index algorithm when dividing the space into grids. When constructing the fractal sequence, the deep representation information is scanned by the Z-order curve paradigm, and the characteristic fractal sequence obtained by the sequence scan is spliced ​​to obtain the fractal sequence representation information. The schematic diagram of the analysis sequence scan is shown in Figure 5 As shown in (d) in the figure, the extracted fractal sequence representation information can also accurately express the dependency relationship between each equally divided pixel block without causing the loss of feature information.

[0051] Deep representation of information through fractal space-filling curve paradigm After serialization processing is performed to obtain the fractal sequence representation information, the feature decoder in the self-supervised learning model is called to perform feature decoding processing on the fractal sequence representation information, thereby obtaining the feature prediction value of the self-supervised learning task. The structure of the feature decoder here is similar to the above feature encoder, both of which include an attention network and a feedforward network, and both have the same number of decoupled fractal attention modules and feedforward modules, so the structure of the feature decoder is not repeated here. Finally, the feedforward network of the feature decoder outputs the feature prediction value for the self-supervised learning task. .

[0052] In the embodiment of the present invention, a fractal sequence is constructed for the extracted deep representation information by using a fractal space filling curve paradigm to obtain fractal sequence representation information for feature decoding to obtain feature prediction values. Therefore, before feature decoding, a fractal space filling curve paradigm is designed to perform fractal serialization construction on the representation information with different information flows. Compared with the prior art that uses a natural language scanning sequence or random serialization to construct a feature sequence, the two-dimensional characteristics between image blocks can be fully considered, and the image representation accuracy can be further improved. In addition, the fractal space filling curve paradigm can be applied to various two-dimensional image scenes and has a certain degree of versatility.

[0053] Step 104: linearly map the feature prediction values ​​to obtain the target prediction result of the self-supervised learning task, and construct the self-supervised learning loss value according to the target prediction result.

[0054] Determining feature prediction values ​​for self-supervised learning tasks After that, a simple output projection layer (such as a fully connected neural network) is used to predict the feature values Linear mapping is performed to obtain the target prediction result of the self-supervised learning task, which is recorded as Y. The target prediction result can be, for example, the classification (recognition) probability of image target recognition, thereby completing the self-supervised learning task of the self-supervised learning model.

[0055] However, when training the self-supervised learning model, the embodiment of the present invention generates a HOG feature map, namely a Histogram of Oriented Gradient (HOG), based on the image to be processed in advance. Then, other pre-trained self-supervised learning models are called to predict and generate the HOG feature map. By executing the corresponding self-supervised learning task, the real label of the image to be processed is finally constructed and recorded as , and mark the true label on the image to be processed. After the self-supervised learning model determines the target prediction result Y of the self-supervised learning task, the true label is determined next. The difference between the predicted result Y and the target result is then used to construct the L2 norm loss function, which is used as the self-supervised learning loss value, denoted as , expressed as the following formula (2): (2)

[0056] In the above formula (2), Indicates calculation of L2 norm.

[0057] The self-supervised learning loss value is calculated by formula (2): Then, we use the self-supervised learning loss value It is used to iteratively train the self-supervised learning model. In each iteration, the self-supervised learning loss value Back propagation is performed in the self-supervised learning model, that is, back propagation is performed in the feature decoder, feature encoder, and image encoder in sequence. The gradient is calculated during the back propagation process and optimized through the gradient descent algorithm. Therefore, each iteration in the back propagation process will optimize the loss value Until the training process reaches the specified number of iterations or loss value The training of the self-supervised learning model ends when it starts to converge (i.e., it stops decreasing). The trained self-supervised learning model can be used for direct application prediction and self-supervised learning tasks.

[0058] The trained self-supervised learning model in the embodiment of the present invention can be applied to various two-dimensional image scenes and has certain performance in different test data sets. Figure 6 , Figure 6 is a schematic diagram of attention visualization analysis on different test data set images provided by the present invention, such as Figure 6 As shown in the figure, multiple different prediction targets are set in a target scene, and there is a certain dependency between the prediction targets, including the fish tank scene dataset on the upper left, the desk scene dataset on the upper right, the bathroom scene dataset on the lower left, and the street scene dataset on the lower right. In various scenes, the trained self-supervised learning model can be used to perform mask image modeling, perform attention calculation on different prediction targets in the scene, and finally obtain the prediction result of the prediction target. Compared with the mask image modeling of the prior art, the trained self-supervised learning model can exert greater feature representation capabilities, is applicable to different target scenes, has universality, and improves the generalization performance of the model.

[0059] In the embodiment of the present invention, the image to be processed for the self-supervised learning task is processed into multiple equally divided pixel blocks, and then the feature encoder is called to perform information flow-based decoupling processing on the equally divided pixel blocks to obtain deep representation information. Next, the fractal space filling curve paradigm is used to construct a fractal sequence for the extracted deep representation information to obtain fractal sequence representation information for feature decoding to obtain feature prediction values. Thus, image features are extracted on the basis of decoupling the information flow between equally divided pixel blocks. Compared with the prior art that does not pay attention to the dependency between image blocks and directly extracts image features at the same time step, image representation can be extracted more accurately. Before feature decoding, a fractal space filling curve paradigm is designed to construct fractal serialization for representation information with different information flows. Compared with the prior art that uses a natural language scanning sequence or random serialization to construct a feature sequence, the two-dimensional characteristics between image blocks can be fully considered, and the image representation accuracy can be further improved. In addition, the fractal space filling curve paradigm can be applied to various two-dimensional image scenes, has a certain versatility, and improves the generalization performance of the model.

[0060] The following describes the self-supervised learning device for mask image modeling based on unified information flow provided by the present invention. The self-supervised learning device for mask image modeling based on unified information flow described below and the self-supervised learning method for mask image modeling based on unified information flow described above can be referenced to each other.

[0061] See also Figure 7 , Figure 7 is a structural diagram of a mask image modeling self-supervised learning device based on unified information flow provided by the present invention, such as Figure 7 As shown, the mask image modeling self-supervised learning device based on unified information flow includes the following modules: an acquisition module 701, a decoupling module 702, a construction module 703, and a training module 704. Among them, the acquisition module 701 is used to acquire the image to be processed for the self-supervised learning task, and call the image block encoder to perform image block processing on the image to be processed to obtain multiple equally divided pixel blocks; the decoupling module 702 is used to call the feature encoder to perform information flow-based decoupling processing on the equally divided pixel blocks to obtain the deep representation information of the image to be processed; the construction module 703 is used to sequence the deep representation information based on the fractal space filling curve paradigm to obtain fractal sequence representation information, and call the feature decoder to perform feature decoding processing on the fractal sequence representation information to obtain the feature prediction value of the self-supervised learning task; the training module 704 is used to linearly map the feature prediction value to obtain the target prediction result of the self-supervised learning task, and construct a self-supervised learning loss value based on the target prediction result, wherein the self-supervised learning loss value is used to train the self-supervised learning model.

[0062] It should be noted that the beneficial effects of the self-supervised learning device for masked image modeling based on a unified information flow here and the self-supervised learning method for masked image modeling based on a unified information flow mentioned above can correspond to each other, so the beneficial effects of the self-supervised learning device for masked image modeling based on a unified information flow will not be repeated here.

[0063] Figure 8 is a schematic diagram of the physical structure of an electronic device provided by the present invention, such as Figure 8 As shown, the electronic device may include: a processor (processor) 810, a communication interface (Communications Interface) 820, a memory (memory) 830 and a communication bus 840, wherein the processor 810, the communication interface 820, and the memory 830 communicate with each other through the communication bus 840. The processor 810 can call the logic instructions in the memory 830 to execute the mask image modeling self-supervised learning method based on the unified information flow, which includes: obtaining an image to be processed for a self-supervised learning task, and calling the image block encoder to perform image block processing on the image to be processed to obtain a plurality of equally divided pixel blocks; calling the feature encoder to perform information flow-based decoupling processing on the equally divided pixel blocks to obtain deep representation information of the image to be processed; based on the fractal space filling curve paradigm, a fractal sequence is constructed for the deep representation information to obtain fractal sequence representation information, and the feature decoder is called to perform feature decoding processing on the fractal sequence representation information to obtain a feature prediction value of the self-supervised learning task; the feature prediction value is linearly mapped to obtain a target prediction result of the self-supervised learning task, and a self-supervised learning loss value is constructed according to the target prediction result, wherein the self-supervised learning loss value is used to train the self-supervised learning model.

[0064] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0065] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the mask image modeling self-supervised learning method based on unified information flow provided by the above methods, the method including: obtaining an image to be processed for a self-supervised learning task, and calling the image block encoder to perform image block processing on the image to be processed to obtain a plurality of equally divided pixel blocks; calling the feature encoder to perform information flow-based decoupling processing on the equally divided pixel blocks to obtain deep representation information of the image to be processed; based on the fractal space filling curve paradigm, performing fractal sequence construction on the deep representation information to obtain fractal sequence representation information, and calling the feature decoder to perform feature decoding processing on the fractal sequence representation information to obtain feature prediction values ​​of the self-supervised learning task; performing linear mapping processing on the feature prediction values ​​to obtain target prediction results of the self-supervised learning task, and constructing a self-supervised learning loss value based on the target prediction results, wherein the self-supervised learning loss value is used to train the self-supervised learning model.

[0066] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the mask image modeling self-supervised learning method based on unified information flow provided by the above-mentioned methods, the method comprising: obtaining an image to be processed for a self-supervised learning task, and calling the image block encoder to perform image block processing on the image to be processed to obtain a plurality of equally divided pixel blocks; calling the feature encoder to perform information flow-based decoupling processing on the equally divided pixel blocks to obtain deep representation information of the image to be processed; based on the fractal space filling curve paradigm, performing fractal sequence construction on the deep representation information to obtain fractal sequence representation information, and calling the feature decoder to perform feature decoding processing on the fractal sequence representation information to obtain feature prediction values ​​of the self-supervised learning task; performing linear mapping processing on the feature prediction values ​​to obtain target prediction results of the self-supervised learning task, and constructing a self-supervised learning loss value based on the target prediction results, wherein the self-supervised learning loss value is used to train the self-supervised learning model.

[0067] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0068] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0069] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A self-supervised learning method for mask image modeling based on unified information flow, characterized in that: Applied to a self-supervised learning model, the self-supervised learning model includes an image block encoder, a feature encoder, and a feature decoder, the method includes: Acquire an image to be processed for a self-supervised learning task, and call the image block encoder to perform image block processing on the image to be processed to obtain a plurality of equally divided pixel blocks; Calling the feature encoder to perform information flow-based decoupling processing on the equally divided pixel blocks to obtain deep representation information of the image to be processed; Based on the fractal space filling curve paradigm, a fractal sequence is constructed for the deep representation information to obtain fractal sequence representation information, and the feature decoder is called to perform feature decoding processing on the fractal sequence representation information to obtain a feature prediction value for the self-supervised learning task; The feature prediction value is linearly mapped to obtain a target prediction result of the self-supervised learning task, and a self-supervised learning loss value is constructed according to the target prediction result, wherein the self-supervised learning loss value is used to train the self-supervised learning model.

2. The mask image modeling self-supervised learning method based on unified information flow according to claim 1, characterized in that: The calling of the image block encoder to perform image block processing on the image to be processed to obtain a plurality of equally divided pixel blocks includes: Performing embedding processing on the image to be processed to obtain a marked embedding sequence; A binary mask sequence corresponding to the image to be processed is obtained, and the label embedding sequence is subjected to image masking by using the binary mask sequence to obtain visible pixel blocks and mask pixel blocks as a plurality of equally divided pixel blocks of the image to be processed.

3. The mask image modeling self-supervised learning method based on unified information flow according to claim 1, characterized in that: The information flow includes a proprioceptive information flow and an exo-body perception information flow; The visible pixel blocks in the equally divided pixel blocks have a combination feature, the combination feature includes an image pixel feature and a pixel position feature, and the mask pixel blocks in the equally divided pixel blocks have a mask position feature; The proprioceptive perception information stream includes a first mapping relationship between combination features or between mask position features; The external body perception information flow includes a second mapping relationship between the combined feature and the mask position feature, or between the mask position feature and the combined feature.

4. The mask image modeling self-supervised learning method based on unified information flow according to claim 3 is characterized in that: Calling the feature encoder to perform information flow-based decoupling processing on the equally divided pixel blocks to obtain deep representation information of the image to be processed, including: When the equally divided pixel blocks have a first mapping relationship, calling the attention network of the feature encoder to perform a self-attention mechanism on the equally divided pixel blocks to obtain attention output information; When the equally divided pixel blocks have a second mapping relationship, calling the attention network of the feature encoder to perform a cross attention mechanism on the equally divided pixel blocks to obtain attention output information; The attention output information is linearly mapped through a feedforward network in the feature encoder to obtain deep representation information of the image to be processed.

5. The mask image modeling self-supervised learning method based on unified information flow according to claim 1, characterized in that: The fractal space filling curve paradigm includes a Hilbert curve paradigm or a Z-order curve paradigm. Based on the fractal space filling curve paradigm, the deep representation information is subjected to fractal sequence construction to obtain fractal sequence representation information, including: The deep representation information is subjected to fractal sequence scanning through the Hilbert curve paradigm, and the characteristic fractal sequences obtained by the sequence scanning are spliced ​​to obtain fractal sequence representation information; or the deep representation information is subjected to fractal sequence scanning through the Z-order curve paradigm, and the characteristic fractal sequences obtained by the sequence scanning are spliced ​​to obtain fractal sequence representation information.

6. The mask image modeling self-supervised learning method based on unified information flow according to claim 1, characterized in that: The image to be processed has a real label, and the real label is constructed according to the HOG feature map generated by the image to be processed. The self-supervised learning loss value is constructed according to the target prediction result, including: Determine the difference between the true label and the target prediction result; An L2 norm loss function is constructed according to the difference, and the L2 norm loss function is used as a self-supervised learning loss value.

7. A mask image modeling self-supervised learning device based on unified information flow, characterized in that: The device comprises: An acquisition module is used to acquire an image to be processed for a self-supervised learning task, and call the image block encoder to perform image block processing on the image to be processed to obtain a plurality of equally divided pixel blocks; A decoupling module, used for calling the feature encoder to perform information flow-based decoupling processing on the equally divided pixel blocks to obtain deep representation information of the image to be processed; A construction module is used to construct a fractal sequence for the deep representation information based on a fractal space filling curve paradigm to obtain fractal sequence representation information, and call the feature decoder to perform feature decoding processing on the fractal sequence representation information to obtain a feature prediction value for the self-supervised learning task; A training module is used to perform linear mapping processing on the feature prediction value to obtain the target prediction result of the self-supervised learning task, and construct a self-supervised learning loss value based on the target prediction result, wherein the self-supervised learning loss value is used to train the self-supervised learning model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the mask image modeling self-supervised learning method based on unified information flow as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the self-supervised learning method for mask image modeling based on unified information flow as claimed in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the self-supervised learning method for mask image modeling based on unified information flow as claimed in any one of claims 1 to 6 is implemented.

Images