Method and apparatus for image characterization learning

Abstract

The invention relates to a method and an apparatus for image characterization learning. An apparatus for image characterization learning, comprising: an encoder for encoding a plurality of image datasets based on a network model to retrieve, for image data in one image dataset of the plurality of image datasets, image data matching another image dataset; and a trainer for training a network model of the encoder to maximize a distance between image data in one image data set and matched image data in the other image data set and to minimize a distance between image data in the one image data set and unmatched image data in the other image data set, wherein the images represented by the matched image data in the plurality of image data sets are obtained at the same time under a plurality of visual angles. Other embodiments are also described.

Description

Technical Field

[0001] The present application generally relates to the fields of computer vision and object detection, and more specifically, to methods and apparatus for learning image representations. Background Art

[0002] Pre-trained models are widely used in the fields of computer vision and object detection. Generally, loading a pre-trained model to learn downstream tasks can achieve higher model accuracy and higher training stability than retraining the entire network. However, existing pre-trained models are not suitable for all datasets. Therefore, a suitable pre-trained model is needed to characterize the corresponding data. Summary of the invention

[0003] According to one aspect of the present application, a device for image representation learning is provided, comprising: an encoder for encoding multiple image data sets based on a network model to retrieve image data matching in another image data set for image data in one image data set among the multiple image data sets; and a trainer for training the network model of the encoder to maximize the distance between the image data in the one image data set and the matching image data in the other image data set and minimize the distance between the image data in the one image data set and the unmatched image data in the other image data set, wherein the images represented by the matching image data in the multiple image data sets are obtained at the same time from multiple perspectives.

[0004] According to another aspect of the present application, a method for image representation learning is provided, comprising: encoding multiple image datasets based on a network model to retrieve image data matching in another image dataset for image data in one of the multiple image datasets; and training the network model to maximize the distance between the image data in the one image dataset and the matching image data in the other image dataset and minimize the distance between the image data in the one image dataset and the unmatched image data in the other image dataset, wherein the images represented by the matching image data in the multiple image datasets are acquired at the same time from multiple perspectives. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] The present invention may be better understood by referring to the description of the embodiments of the present invention in conjunction with the following drawings, wherein like reference numerals indicate the same or functionally similar elements:

[0006] Figure 1 According to an embodiment of the present application, a schematic diagram of a transmission device with adjustable object posture is shown.

[0007] Figure 2 A schematic diagram of a device for image representation learning is shown according to an embodiment of the present application.

[0008] Figure 3 According to an embodiment of the present application, a schematic diagram of a multi-view image representation learning contrast learning algorithm is shown.

[0009] Figure 4 According to an embodiment of the present application, a flowchart of a method for image representation learning is shown.

[0010] Figure 5 A schematic diagram of the structure of an information processing device that can implement the robot or remote control device in the embodiments of the present application is shown. DETAILED DESCRIPTION

[0011] The features and exemplary embodiments of various aspects of the present invention will be described in detail below. In the detailed description below, many specific details are proposed to provide a comprehensive understanding of the present invention. However, it is obvious to those skilled in the art that the present invention can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present invention by illustrating examples of the present invention. The present invention is by no means limited to any specific configuration and algorithm proposed below, but covers any modification, replacement and improvement of elements, parts and algorithms without departing from the spirit of the present invention. In the accompanying drawings and the following description, known structures and technologies are not shown to avoid unnecessary ambiguity to the present invention.

[0012] Pre-trained models are widely used in the fields of computer vision and object detection. Generally, loading a pre-trained model to learn downstream tasks can achieve higher model accuracy and higher training stability than retraining the entire network, because the pre-trained model has learned data representations with strong generalization in the proxy task.

[0013] The pre-trained models commonly used in object detection are trained based on large-scale image classification datasets (e.g., ImageNet series datasets) or large-scale object detection datasets (e.g., MSCOCO series datasets). However, these data themselves have relatively large domain differences from some data (e.g., image data irradiated by radiation such as X-rays). For example, X-ray images have characteristics such as perspective and distortion, while the above datasets have almost no or little involvement in these attributes. Therefore, the existing pre-trained models lack modeling and description of some special attributes in the representation of data such as X-ray data.

[0014] The above method of using a pre-trained model in one field as a starting point to redevelop a new model for a task in another field is a specific form of transfer learning. In the classic machine learning method of transfer learning, there is a set of concepts: source domain and target domain: the domain where the data used to train the pre-trained model is located is called the source domain; and the domain where the task to be developed is located is called the target domain. When there is a large difference between the data in the target domain and the source domain, directly migrating the source domain feature representation to the target domain will result in a large loss of accuracy, and training pre-trained models suitable for this field by designing proxy tasks has become a direction that is gaining more and more attention. Technical solutions for designing proxy tasks can include image generation, image restoration, contrastive learning, etc.

[0015] Image generation tasks obtain generative models by learning the probability distribution of the observed data itself, such as generative adversarial networks, (variational) autoencoders, etc. The output of a specific layer of the network can be considered as a feature representation. However, compared with other proxy tasks, image generation tasks have disadvantages such as large model capacity and unstable training.

[0016] The image restoration task randomly generates a mask on the image to erase the corresponding content in advance, and the training model uses the image context information as input to restore the information covered by the mask. However, in scenarios where the correlation of image context information is not high (for example, objects are placed in disorder in X-ray security inspection scenarios), the image restoration task is not applicable.

[0017] In contrastive learning, we first need to introduce the concepts of positive and negative samples. Positive samples refer to samples of the same type, while negative samples refer to samples of different types. Here, the so-called same type and different types can be pre-defined manually. Positive and negative samples can be combined in pairs to construct multiple groups of positive sample pairs and negative sample pairs, which can be referred to as positive pairs and negative pairs. Contrastive learning trains the encoder (or model or network) by constructing positive and negative samples, requiring that in the encoded feature space, the representations of positive samples are close to each other, while the representations of negative samples are far apart.

[0018] The significant difference between different contrastive learning methods lies in the way the positive and negative samples are constructed. A common contrastive learning proxy task is called instance discrimination, which is to determine whether a group of samples are the same instance. Two samples of the same instance are positive pairs, and two samples of different instances are negative pairs. Similar to the difference between instance segmentation and semantic segmentation, different instances may belong to the same category to some extent semantically.

[0019] The instance discrimination task in the image field can usually be designed as follows: take an input image as an instance, and perform multiple (for example, two) different data augmentations on it. The augmented images are called views. Different views from the same instance are positive pairs, and views from different instances are negative pairs. When a view is given as a query, the proxy task requires the encoder to encode all views, and then use a simple distance metric (for example, cosine similarity, etc.) in its feature space to match another view of the same instance in a dictionary containing a large number of negative pairs. When the encoder model trained by this proxy task can handle the pairing task well, it can be considered to have good data representation capabilities, that is, the representation distance between the same instance is close and the representation distance between different instances is far.

[0020] The key to the above contrastive learning proxy task is how to design appropriate positive and negative pairs for training, which depends on the choice of data augmentation method. Data augmentation can be strong or weak, and different degrees of data augmentation have an important impact on the training of the model and the representation ability of the model: weak data augmentation may make the task too simple and have trivial solutions, resulting in the model trained by the proxy task having a weak representation ability of the data; strong data augmentation may make the task too difficult and the model may not converge during training. In practical applications, how to construct appropriate data augmentation requires a lot of hyperparameter searches, which is very labor-intensive and computationally intensive.

[0021] The present application recognizes that in multi-perspective scenarios such as X-ray security inspections, multiple image acquisition devices (e.g., multiple groups of radiation sources and radiation detectors) can simultaneously generate multi-perspective images such as looking down, looking sideways, and looking up, and therefore have the function of constructing different views of the same instance. For example, the same instance is a test object at a certain moment, and its different views are the pictures of the test object at this moment collected by the item detection device. In addition to the radiation images collected by multi-eye radiation source emitters and detectors, images of two or more different perspectives (e.g., visible light images, depth images, infrared images, etc.) recorded by multi-eye cameras at the same time can also construct different views of the same instance, and the present application does not limit this. Simple data augmentation (e.g., cropping and scaling, etc.) can be used to process the pictures of looking down, looking sideways, and looking up.

[0022] Figure 1 According to an embodiment of the present application, a schematic diagram of a transport device with adjustable object posture is shown. As shown in the figure, the transport device can make the object move parallel to the coordinate axes such as x, y, and z, for example, move horizontally along the x-axis, move up and down along the y-axis, and move forward and backward along the z-axis. The transport device can also make the object rotate along the coordinate axes such as x, y, and z, for example, Figure 1The rotation along the y-axis is shown in the figure, and the rotation along the x-axis and the z-axis (not shown) is also possible, and the present application is not limited in this respect.

[0023] In some embodiments, the transport device may also include an article machine crawler for transporting articles, such as Figure 1 shown.

[0024] Through the transmission device with adjustable object posture, in conjunction with multiple image acquisition devices in the radio frequency device (for example, multiple groups of radiation sources and radiation detectors), a large-scale multi-view image set can be collected and constructed.

[0025] Figure 2 According to an embodiment of the present application, a schematic diagram of an apparatus 200 for image representation learning is shown. As shown in the figure, the apparatus 200 may include an encoder 210 and a trainer 220. In some embodiments, the encoder 210 may encode multiple image data sets based on a network model to retrieve image data matching another image data set for image data in one of the image data sets. The image data is used to characterize the image, thereby, a matching image can be obtained. The trainer 220 may train the network model of the encoder to maximize the distance between the image data in the one image data set and the matching image data in the other image data set and minimize the distance between the image data in the one image data set and the unmatched image data in the other image data set. The above-mentioned network model includes a network structure and / or network parameters, and may also include other aspects of the network model, which is not limited by the present application. The images represented by the matching image data in the above-mentioned multiple image data sets are obtained at the same time from multiple perspectives, for example, by Figure 1 The images obtained by the transmission device with adjustable object posture at different perspectives at the same time stamp are matched, in other words, they belong to the positive alignment described above.

[0026] In some embodiments, the image data in the image data set may be a pixel value or a feature representation obtained in a certain manner, and this application does not impose any restrictions on this.

[0027] In some embodiments, the distance between the image data may be represented by cosine similarity. In other embodiments, the distance between the image data may be represented by other methods, which are not limited in the present application.

[0028] Figure 3 According to an embodiment of the present application, a schematic diagram of a multi-view image matching contrast learning algorithm is shown. Figure 3 , an example of two image data sets being encoded by two encoders respectively is shown. As shown in the figure, the image data set under the first viewing angle is encoded by encoder 1 to generate I 1 to IN ; The image data set under the second perspective is encoded by encoder 2 to generate P 1 To P N The trainer can train encoder 1 and encoder 2 jointly or separately, with the goal of making P 1 I 1 To P N I N The matrix diagonal matching result. For example, the matrix diagonal elements P 1 I 1 To P N I N The value of is 1 (indicating a match), and the other elements are 0 (indicating a mismatch).

[0029] Figure 3 The number of the image data sets shown is two, for example, two image data sets obtained via two viewing angles. However, in some embodiments, the number of the plurality of image data sets may be more than two, and the present application is not limited in this respect.

[0030] In some embodiments, taking an image dataset obtained under two perspectives as an example, given a batch of N first-perspective and second-perspective image pairs, the corresponding encodings are obtained by encoder 1 and encoder 2 respectively, and the above two encoders are jointly trained to learn the same embedding space to maximize the cosine similarity of the N correctly matched multi-perspective image embeddings in the batch, while minimizing the N correctly matched multi-perspective image embeddings in the batch. 2 - Cosine similarity of N incorrectly matched image embeddings.

[0031] In some embodiments, the trainer may train the network model based on a loss function (or penalty function) as an objective function (e.g., continuously updating the network model until a better or optimal network model is obtained). For example, the trainer may train the encoder model based on minimizing the loss function. In addition, the trainer may also train other learnable modules associated with the encoder (e.g., a projector, a normalizer, etc.). The following description is given by taking an image data set obtained under two viewing angles as an example.

[0032] A batch of data is randomly sampled as input, including N sets of paired images from the first and second perspectives (x v , x u ), the first view and the second view feature representation pair (v, u) are encoded by the corresponding encoder. The subscript i can be used to represent the i-th pair, then the contrast loss from v to u of the pair of data can be expressed as follows:

[0033]

[0034] in, <v i,u i > represents cosine similarity, for example,<v,u> =v T u / ||v||||u||, or τ is a positive real number representing a temperature hyperparameter, which can be predefined or adjusted based on the training of the trainer. The loss function (1) has a similar form to the InfoNCE loss, and minimizing this loss is equivalent to maximizing the mutual information of the encoder's true pairwise feature representations. In some embodiments, the loss function can be a multi-pair classification loss, where the classification task can be described as predicting v from a set of data. i Real pairing u i .

[0035] Similarly, the contrast loss from u to v for the i-th pair of data is:

[0036]

[0037] The final loss function can be the weighted average of the above two losses for each batch of N pairs of data:

[0038]

[0039] Here, λ is a scalar ranging from 0 to 1. For example, λ=0.5 may be set.

[0040] The larger the value of the final loss function, the greater the adjustment that the trainer needs to make to the encoder model and other learnable modules. The smaller the value of the final loss function, the smaller the adjustment that the trainer needs to make to the encoder model and other learnable modules. Theoretically, when the final loss function is 0, the obtained encoder model and other learnable modules can be used as the final trained encoder model and other learnable modules. The output of any one or more network layers of such encoder models and other learnable modules can be used as data representation in downstream tasks, for example, the encoder can be combined with a detection head for target detection tasks, etc.

[0041] The design of the above loss function is only an example. Based on the principles of this application, other loss functions can be designed to train the encoder model and other associated learnable modules to obtain an encoder that can correctly match the image data, and this application does not limit this. In addition, the above loss function only takes the image data set obtained from two perspectives as an example, and can be expanded to a loss function applicable to image data sets obtained from more than two perspectives based on the principles of this application, and this application does not limit this.

[0042] In some embodiments, the apparatus 200 for image representation learning may further include a data augmenter for performing data augmentation on multiple original data sets to generate multiple image data sets. Data augmentation may include, for example, cropping, scaling, or other augmentation methods, which are not limited in this application.

[0043] In some embodiments, the apparatus 200 for image representation learning may further include a projector. After encoding multiple image data sets, the encoder outputs multiple feature representation sets (e.g., a first feature representation set), and the projector is used to project the multiple feature representation sets to obtain multiple projected feature representation sets (e.g., a second feature representation set), thereby achieving dimensionality increase or dimensionality reduction of image embedding. The trainer 220 can train the encoder model and other learnable modules based on the projected multiple feature representation sets. For example, the feature representation involved in the cosine similarity calculation is after projection. The projector can be linear or nonlinear. As one of the modules associated with the encoder, the projector can have learnable parameters for mapping the data representation obtained by the encoder to the same embedding space, so as to facilitate the distance measurement, loss function calculation, etc. between subsequent data representations. The trainer can update the learnable parameters of the projector based on the projected data representation.

[0044] In some embodiments, the structure of the encoder network model may include a deep neural network, such as a convolutional neural network (CNN) or a vision transformer (ViT). In some embodiments, the structure of the encoder network model may include other models or networks, which is not limited in this application.

[0045] In some embodiments, the dense feature representation of the image after being encoded by the encoder can be converted into a global feature representation, that is, reduced from a tensor to a vector. For example, a pooling operation can be performed on the output of the CNN encoder. The pooling operation may include attention pooling or global average pooling, etc. For example, the output of the ViT encoder is usually a sequence of multiple vectors, and a vector in the sequence can be taken as a global feature representation. This application is not limited to this.

[0046] In some embodiments, the image datasets under each viewing angle may use the same encoder model (e.g., the same network structure and network parameters). In some embodiments, the image datasets under different viewing angles use different encoder models (e.g., different network structures and / or different network parameters). When the network structures and network parameters of two model instances are exactly the same, the two model instances are the same; when one or both of the network structures and network parameters of the model are different, the two model instances are said to be different.

[0047] Two examples are provided below.

[0048] In the first example, two image encoders (e.g., CNN networks such as resnet-50 networks, or ViT networks) are randomly initialized as image encoders for the first view and the second view, respectively, and the dimensions of their output feature representations can be set, for example, to 512. Each batch samples N (e.g., 1024) pairs of first-view and second-view training pictures from the training set. The first-view picture passes through the first-view image encoder, for example, to obtain a feature matrix with a shape of [1024,512]; the second-view picture passes through the second-view image encoder, for example, to obtain a feature matrix with a shape of [1024,512]. Multiply the first-view feature matrix by the transpose of the second-view feature matrix, for example, to obtain a cosine similarity matrix of [1024,1024]. The network parameters of the image encoder are updated by back-propagation of the gradient of the symmetric cross entropy loss to obtain the final image encoder model.

[0049] In the second example, an image encoder (e.g., a ViT network, or a CNN network) is randomly initialized as an image encoder shared by the first and second views, and the dimension of its output feature representation is set to, for example, 768. Each batch samples N (e.g., 1024) pairs of first-view and second-view training pictures from the training set. The first-view image and the second-view picture are respectively passed through the image encoder, for example, to obtain two feature matrices of shape [1024,768]. The first-view feature matrix is ​​multiplied by the transpose of the second-view feature matrix, for example, to obtain a cosine similarity matrix of [1024,1024]. The network parameters of the image encoder are updated by symmetric cross entropy loss gradient back propagation to obtain the final image encoder model.

[0050] In the first example, different viewpoints use different image encoders and the network parameters are not shared, thereby meeting the need for separate predictions for different viewpoints. In the second example, different viewpoints share the same image encoder and the network parameters are shared, thereby meeting the need for view-independent predictions using a common model.

[0051] Figure 4According to an embodiment of the present application, a flow chart of a method 400 for image representation learning is shown. The method 400 may include steps 410 and 420.

[0052] At 410, multiple image datasets are encoded based on the network model to retrieve image data matching another image dataset for image data in one of the multiple image datasets. The images represented by the matching image data in the multiple image datasets may be obtained at the same time under multiple viewing angles.

[0053] At 420 , the network model is trained to maximize the distance between image data in one image dataset and matching image data in another image dataset and to minimize the distance between image data in one image dataset and unmatched image data in another image dataset.

[0054] Other embodiments of the method 400 may be understood by referring to the above description, and will not be described in detail here.

[0055] By using the method and apparatus described in the present disclosure, data representations suitable for images (e.g., radio frequency radiation images such as X-ray images) are learned through comparative learning tasks, which can better describe the potential attributes of a specific image, such as perspective and distortion. The pre-trained model trained by this method is used as the backbone network of downstream tasks (e.g., feature extractor module), which can effectively improve the accuracy of downstream tasks (e.g., target detection, semantic segmentation, instance segmentation, etc.). In addition, compared with the model training method of classification labeling, the method described in the present disclosure can avoid manual labeling, thereby improving training efficiency.

[0056] Figure 5 FIG. 5 shows a schematic diagram of the structure of an information processing device 500. The robot and / or remote monitoring device in the embodiment of the present application can be implemented by the information processing device 500. Figure 5 As shown, device 500 may include one or more of the following components: a processor 520 , a memory 530 , a power component 540 , an input / output (I / O) interface 560 , and a communication interface 580 , which components may be communicatively connected via a bus 510 , for example.

[0057] The processor 520 controls the operation of the device 500 as a whole, such as operations associated with data communication and computing processing. The processor 520 may include one or more processing cores and may execute instructions to implement all or part of the steps of the method described in the present application. The processor 520 may include various devices with processing functions, including but not limited to general-purpose processors, special-purpose processors, microprocessors, microcontrollers, graphics processors (GPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), etc. The processor 520 may include a cache 525 or may communicate with the cache 525 to increase the access speed of data.

[0058] The memory 530 is configured to store various types of instructions and / or data to support the operation of the device 500. Examples of data include instructions, data, etc. for any application or method operating on the device 500. The memory 530 can be implemented by any type of volatile or non-volatile storage device or a combination thereof. The memory 530 can include semiconductor memory, such as random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, etc. The memory 530 can also include, for example, any memory using paper media, magnetic media and / or optical media, such as paper tape, hard disk, magnetic tape, floppy disk, magneto-optical disk (MO), CD, DVD, Blue-ray, etc.

[0059] Power supply assembly 540 provides power to the various components of device 500. Power supply assembly 540 may include an internal battery and / or an external power interface, and may include a power management system and other components associated with generating, managing, and distributing power for device 500.

[0060] The I / O interface 560 provides an interface that enables a user to interact with the device 500. The I / O interface 560 may include, for example, interfaces based on technologies such as PS / 2, RS-232, USB, FireWire, Lightening, VGA, HDMI, DisplayPort, etc., so that a user can interact with the device 500 through peripheral devices such as a keyboard, a mouse, a touch pad, a touch screen, a joystick, a button, a microphone, a speaker, a display, a camera, a projection port, etc.

[0061] The communication interface 580 is configured to enable the device 500 to communicate with other devices in a wired or wireless manner. The device 500 can access a wireless network based on one or more communication standards, such as Wi-Fi, 2G, 3G, 4G, and 5G communication networks, through the communication interface 580. In an exemplary embodiment, the communication interface 580 can also receive a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. The exemplary communication interface 580 may include an interface based on communication methods such as near field communication (NFC) technology, radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, and Bluetooth (BT) technology.

[0062] The functional blocks shown in the above-described block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present invention are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link by a data signal carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segment can be downloaded via a computer network such as the Internet, an intranet, etc.

[0063] The present invention can be implemented in other specific forms without departing from its spirit and essential features. For example, the algorithms described in the specific embodiments can be modified, and the system architecture does not depart from the basic spirit of the present invention. Therefore, the current embodiments are regarded as exemplary and non-restrictive in all aspects, and the scope of the present invention is defined by the appended claims rather than the above description, and all changes falling within the meaning and equivalent scope of the claims are thus included in the scope of the present invention.

Claims

1. A device for image representation learning, comprising: An encoder for encoding a plurality of image datasets based on a network model to retrieve image data matching another image dataset for image data in one of the plurality of image datasets; as well as a trainer for training the network model of the encoder to maximize the distance between the image data in the one image dataset and the matching image data in the other image dataset and to minimize the distance between the image data in the one image dataset and the unmatched image data in the other image dataset, The images represented by the matching image data in the multiple image data sets are acquired at the same time under multiple viewing angles.

2. The device according to claim 1, wherein: The trainer is used to: The network model is trained based on the loss function.

3. The apparatus of claim 1, further comprising: The data augmentor is used to perform data augmentation on a plurality of original data sets to generate the plurality of image data sets.

4. The device according to claim 1, wherein: The plurality of image data sets are encoded by the encoder to obtain a plurality of feature representation sets, and the device further comprises: a projector, configured to project the plurality of feature representation sets to obtain a plurality of projected feature representation sets, The trainer is used to train the parameters of the network model and the projector based on the projected multiple feature representation sets.

5. The device according to claim 4, wherein: The projector is linear or non-linear.

6. The device according to claim 1, wherein: The network structure of the network model includes a convolutional neural network (CNN) or a visual transformer (ViT).

7. The device of claim 1, wherein: The network models are the same or different for the multiple image datasets.

8. The device of claim 1, wherein: The image data in the plurality of image data sets are obtained via a device whose posture can be adjusted in multiple dimensions, and the device can move parallely or rotate along a coordinate axis.

9. The device of claim 8, wherein: The device is arranged with a plurality of image acquirers.

10. The device of claim 1, wherein: The image data in the plurality of image data sets include data of images after being irradiated with radiation.

11. A method for image representation learning, comprising: encoding the plurality of image datasets based on the network model to retrieve, for image data in one of the plurality of image datasets, image data that matches image data in another image dataset; as well as The network model is trained to maximize the distance between the image data in the one image dataset and the matching image data in the other image dataset and to minimize the distance between the image data in the one image dataset and the unmatched image data in the other image dataset, The images represented by the matching image data in the multiple image data sets are acquired at the same time under multiple viewing angles.

12. The method of claim 11, further comprising: The network model is trained based on the loss function.

13. The method of claim 11, further comprising: Data augmentation is performed on a plurality of original data sets to generate the plurality of image data sets.

14. The method of claim 11, wherein: After encoding the multiple image data sets, multiple feature representation sets are obtained. The method further includes: Projecting the plurality of feature representation sets to obtain a plurality of projected feature representation sets; and The network model and parameters used by the projection are trained based on the projected multiple feature representation sets.

15. The method of claim 14, wherein: The projection is linear or non-linear.

16. The method of claim 11, wherein: The network structure of the network model includes a convolutional neural network (CNN) or a visual transformer (ViT).

17. The method of claim 11, wherein: The network models are the same or different for the multiple image datasets.

18. The method of claim 11, wherein: The image data in the plurality of image data sets are obtained via a device whose posture can be adjusted in multiple dimensions, and the device can move parallely or rotate along a coordinate axis.

19. The method of claim 18, wherein: The device is arranged with a plurality of image acquirers.

20. The method of claim 11, wherein: The image data in the plurality of image data sets include data of images after being irradiated with radiation.

Images