Zero sample classification method and device for geoscience literature pictures and electronic equipment

Through the Geo-CLIP model, the geographic pictures and text descriptions in geographic literature are mapped to the same feature space, which solves the problem that geographic literature image classification requires a large amount of labeled data, and achieves the classification effect of high accuracy and generalization performance.

CN119006868APending Publication Date: 2024-11-22CHINA PETROLEUM & CHEMICAL CORP +1
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
CN202310562960.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-18
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

The prior art requires a large amount of labeled data in the classification of geological literature pictures, which leads to the accuracy of labeled data affecting the classification effect. The generalization performance of the supervised learning method is poor, making it difficult to effectively and accurately classify geological literature pictures.

Method used

The Geo-CLIP model is used to map the geographic pictures and text description data in geographic literature to the same feature space. Through comparative learning and training models, the text description is generated to realize image classification, avoiding the need for large-scale annotation of data.

Benefits of technology

It realizes the accurate classification of geologic literature pictures, reduces the dependence of artificial labeled data, improves the generalization performance and classification accuracy of the model, and is suitable for picture recognition of geologic research literature.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119006868A_ABST
    Figure CN119006868A_ABST
Patent Text Reader

Abstract

According to the geoscience literature picture zero sample classification method and device and the electronic equipment provided by the embodiment of the invention, data pairs of geographic pictures and text description are analyzed from geoscience literatures, and the text description is a description statement corresponding to the geographic pictures in the geoscience literatures; obtaining an image feature vector and a text feature vector according to the data pair; the image feature vector is used as a transverse input vector, the text feature vector is used as a longitudinal input vector, N positive sample pairs and N (N-1) negative sample pairs are obtained, the N positive sample pairs and the N (N-1) negative sample pairs are used for training a pre-established Geo-CLIP model, and N is the number of geographic pictures; and inputting a to-be-identified literature picture into the Geo-CLIP model, and generating a text description according to the to-be-identified literature picture, the text description representing a classification result of the literature picture. And large-scale data labeling is not needed, so that the problem that subsequent geoscience literature picture recognition is inaccurate due to inaccurate data labeling is avoided.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and specifically to a method, device and electronic equipment for zero-sample classification of geological literature images. Background Art

[0002] In the study of understanding and classification of geoscience literature images, it is necessary to parse the PDF documents first, and then conduct in-depth analysis of the parsed text, images, tables and other data, such as scientific research literature image classification. At present, the parsing technology of PDF files has been relatively mature. For example, structured parsing tools such as Science Parse, PDFFigures 2.0 and Grobi can easily extract the image and text information in PDF documents.

[0003] In the related research on image classification, the supervised learning-based method has achieved good results. In the Chinese invention patent application with application number CN202210350617.0, a venue recognition method, system, device and storage medium based on multimodality are proposed. The invention can use multimodal retrieval based on a large number of online pictures of the guide to realize an automatic recognition mechanism for open scenes, greatly reducing labor costs and facilitating subsequent applications in other scenes. In the Chinese invention patent application with application number CN202111465039.7, an image search method and device based on a multimodal algorithm are proposed. This method uses a multimodal algorithm to realize a fast and accurate image search method, but additionally uses a database to store multimodal features, which is not very useful in large-scale multi-classification image recognition. In the Chinese invention patent application with application number CN201910836905.5, a zero-sample deep-sea biological image classification method based on multiple classifiers is proposed. This invention reduces the influence of mapping domain drift to a certain extent, is accurate and easy to implement, but this method still needs to construct an image feature dataset, which is not applicable in the case of geoscience literature images. In the Chinese invention patent application with application number CN201910449243.6, a method for extracting scientific and technological literature images based on Faster-RCNN is proposed. This invention has fast detection speed and high accuracy, which helps to further analyze and understand scientific and technological literature images and has high practical application value. However, it is necessary to build a data set to train the convolutional neural network model, and it does not support image classification, so it cannot be used to realize image recognition of geoscience research literature.

[0004] From the above, it can be seen that the success of existing supervised learning methods in image classification tasks is inseparable from two key points: a large amount of labeled data and large model training. However, image classification based on supervised learning uses labeled data with given labels for model training to classify new samples in a set of predetermined object categories. This limited form of supervision limits their versatility and usability, and also makes the generalization performance of this method poor. Second, supervised learning requires a large amount of manually labeled data to participate in model training in order to achieve better results. A large amount of labeled data means that a lot of manpower and material resources need to be invested in labeling. However, labeling in professional fields such as geology has certain requirements for the professional knowledge of the participants, which also leads to an extreme lack of high-quality labeled data in scientific research literature, especially in earth science literature, making it difficult to effectively and accurately classify images in earth science literature. Summary of the invention

[0005] In view of this, the present application provides a zero-sample classification method, device and electronic device for document images, so as to solve the problem in the prior art that the classification of geological document images requires a large amount of labeled data, and the accuracy of the labeled data affects the classification effect of geological document images.

[0006] In a first aspect, the present application provides a method for zero-sample classification of geoscience literature images, including:

[0007] Parsing data pairs of geographic images and text descriptions from geoscience documents, wherein the text descriptions are description sentences corresponding to the geographic images in the geoscience documents;

[0008] Obtaining an image feature vector and a text feature vector according to the data pair;

[0009] The image feature vector is used as a horizontal input vector, and the text feature vector is used as a vertical input vector to obtain N positive sample pairs and N (N-1) negative sample pairs, and the N positive sample pairs and N (N-1) negative sample pairs are used to train a pre-established Geo-CLIP model, where N is the number of geographic images;

[0010] The document image to be identified is input into the Geo-CLIP model, and the Geo-CLIP model generates a text description according to the document image to be identified, wherein the text description represents a classification result of the document image.

[0011] In a possible implementation, the step of parsing data pairs of geographic images and text descriptions from geographic literature includes:

[0012] Parsing the geography document by PDF to determine (geographical image, text description) pairs, wherein the text description in each (geographical image, text description) pair corresponds to the geographic image in the current pair;

[0013] The geographic image and the text description in the (geographic image, text description) pair are separated.

[0014] In a possible implementation, the Geo-CLIP model includes a picture editor and a text editor, and obtaining an image feature vector and a text feature vector according to the data pair includes:

[0015] Inputting the geographic picture in the geography document into the picture editor and mapping it into the first feature space to obtain a first image feature vector;

[0016] The text description in the geoscience document is input into the text editor and mapped to the second feature space to obtain a first text feature vector, and the number of the text feature vector is the same as that of the image feature vector.

[0017] In a possible implementation manner, the feature vectors output by the image encoder and the text encoder have different lengths, and an additional linear mapping is used to map the first image feature vector and the first text feature vector to a uniform length.

[0018] In a possible implementation, a loss function consisting of N positive sample pairs and N(N-1) negative sample pairs is used, and then the Geo-CLIP model is trained by contrastive learning.

[0019] In a possible implementation, the inputting of the document image to be identified into the Geo-CLIP model, and the Geo-CLIP model generating a text description according to the document image to be identified, includes:

[0020] Mapping the document image to be identified through an image encoder to obtain a second image feature vector;

[0021] Map the candidate category labels through the text encoder to obtain a text feature vector group;

[0022] A second text feature vector in the text feature vector group having the greatest similarity to the second image feature vector is determined, and a text description of the second text feature vector is the classification result of the document image to be identified.

[0023] In a possible implementation manner, determining the second text feature vector in the text feature vector group that has the greatest similarity to the second image feature vector includes:

[0024] Comparing each text feature vector of the text feature vector group with the second image feature vector to obtain a confidence value;

[0025] The text feature vector with the highest confidence value is used as the classified text feature vector of the second image feature vector.

[0026] In a second aspect, the embodiment of the present application provides a zero-sample classification device for geoscience literature images, including:

[0027] A parsing module, used to parse out data pairs of geographical images and text descriptions from geographical documents, wherein the text descriptions are description sentences corresponding to the geographical images in the geographical documents;

[0028] An acquisition module, used for acquiring an image feature vector and a text feature vector according to the data pair;

[0029] A model training module, used to obtain N positive sample pairs and N (N-1) negative sample pairs by using the image feature vector as a horizontal input vector and the text feature vector as a vertical input vector, wherein the N positive sample pairs and the N (N-1) negative sample pairs are used to train a pre-established Geo-CLIP model, where N is the number of geographic images;

[0030] The image classification module is used to input the document image to be identified into the Geo-CLIP model, and the Geo-CLIP model generates a text description based on the document image to be identified, and the text description represents the classification result of the document image.

[0031] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0032] processor;

[0033] Memory;

[0034] And a computer program, wherein the computer program is stored in the memory, and the computer program includes instructions, which, when executed by the processor, enable the electronic device to execute the method described in any possible implementation of the first aspect.

[0035] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored program, wherein when the program is running, the device where the computer-readable storage medium is located is controlled to execute the method described in any possible implementation method of the first aspect.

[0036] In the embodiment of the present application, the image and text feature vectors of the training model are determined by using the geological literature file itself, the established model is trained, and then the trained model is used to determine the classification result according to the literature image. There is no need to label data on a large scale, thus avoiding the problem of inaccurate recognition of subsequent geological literature images due to inaccurate labeling data. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0038] Figure 1 A flowchart of a zero-sample classification method for geoscience literature images provided in an embodiment of the present application;

[0039] Figure 2 A schematic diagram of generating training samples provided in an embodiment of the present application;

[0040] Figure 3 A schematic diagram of the classification results of geoscience literature images provided in the embodiment of the present application;

[0041] Figure 4 Verification of geoscience literature images provided for the embodiments of this application;

[0042] Figure 5 Schematic diagram of a zero-sample classification device for geoscience literature images provided in an embodiment of the present application

[0043] Figure 6 A schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to better understand the technical solution of the present application, the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0045] It should be clear that the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0046] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0047] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0048] like Figure 1 As shown, the zero-sample classification method for geoscience literature images in this embodiment includes:

[0049] S101, parsing data pairs of geographical images and text descriptions from geographical documents, wherein the text descriptions are description sentences corresponding to the geographical images in the geographical documents.

[0050] The embodiment of the present application classifies the images parsed from the geoscience literature by labeling them, so as to make full use of the large-scale image data to extract richer information, complete the construction of the multimodal scientific database, and drive the research of geoscience big data. Specifically, with the help of CLIP and Chinese CLIP multimodal pre-training models, Geo-CLIP is developed on this basis to realize the image understanding and classification of geoscience academic literature, forming a zero-sample classification method for scientific literature images.

[0051] Specifically, in this embodiment, the geological document is subjected to PDF parsing to determine (geographical image, text description) pairs, wherein the text description in each (geographical image, text description) pair corresponds to the geographical image in the current pair. The geographical image and text description in the (geographical image, text description) pair are split. For example, if there is a picture of the Himalayas in the geological document, the text description corresponding to the picture of the Himalayas is parsed, and the two constitute a (geographical image, text description) pair.

[0052] S102: Obtain an image feature vector and a text feature vector according to the data pair.

[0053] Geo-CLIP consists of two encoders: an image encoder and a text encoder. The geographic image in the geoscience document is input into the image editor and mapped to the first feature space to obtain a first image feature vector; the text description in the geoscience document is input into the text editor and mapped to the second feature space to obtain a first text feature vector, and the number of the text feature vector and the image feature vector is the same.

[0054] The core idea of ​​Geo-CLIP is to map geography-related images and their corresponding text descriptions in geological literature into the same feature space. This feature space is an abstract concept. For example, when you see a picture of the "Himalayas", it is mapped to the concept of the Himalayas. When you read the text of the "Himalayas", it is also mapped to the concept of the Himalayas. Therefore, this concept can be regarded as the "feature space" of the Himalayas.

[0055] It should be noted that since the feature vectors output by the image encoder and the text encoder have different lengths, Geo-CLIP uses an additional linear mapping to map the feature vectors generated by the two encoders to a uniform length.

[0056] S103, respectively using the image feature vector as a horizontal input vector and the text feature vector as a vertical input vector to obtain N positive sample pairs and N(N-1) negative sample pairs, wherein the N positive sample pairs and N(N-1) negative sample pairs are used to train a pre-established Geo-CLIP model, where N is the number of geographic images.

[0057] During model training, a batch of N (geographic images, text description) pairs is taken each time. Figure 2 , N geographic images are fed into the image encoder to obtain N image feature vectors (I1, I2, ..., I N ), and at the same time, N text descriptions are sent to the text encoder to obtain N text feature vectors (T1, T2, ..., T N ). Since only geographic images and text descriptions on the diagonal are a pair, the training goal of Geo-CLIP is to make the feature vector similarity of a (geographic image, text description) pair as high as possible, rather than the similarity of a pair as low as possible. In this way, the Geo-CLIP model constructs a loss function consisting of N positive sample pairs and N(N-1) negative sample pairs, and then trains the model through contrastive learning.

[0058] S104, inputting the document image to be identified into the Geo-CLIP model, and the Geo-CLIP model generates a text description according to the document image to be identified, wherein the text description represents the classification result of the document image.

[0059] After training the model, Geo-CLIP has the ability to map images and texts in geoscience literature to the same feature space. For geoscience academic literature image classification applications, the image to be identified is input into the trained model, and Geo-CLIP can automatically generate the text description corresponding to the image.

[0060] When classifying geological literature images, the literature image to be identified is mapped through an image encoder to obtain a second image feature vector; the candidate category label is mapped through a text encoder to obtain a text feature vector group; the second text feature vector in the text feature vector group that has the greatest similarity to the second image feature vector is determined, and the text description of the second text feature vector is the classification result of the literature image to be identified.

[0061] like Figure 3 As shown in the figure, the document image to be identified is a picture of a bridge, and the second image feature vector I1 is obtained by mapping the image encoder. The candidate labels river, mountain, bridge, ... lake are mapped through the text encoder to obtain a text feature vector group (T1, T2, ..., T N), and finally determine that the second text feature vector in the text feature vector group with the greatest similarity to the second image feature vector is T3, that is, the classification result of the document image to be identified is bridge.

[0062] In order to verify the accuracy of zero-sample classification of geoscience literature images, this example Figure 4 The pictures in were verified. Figure 4 In (a), if the candidate labels are: [mountains, rivers, oceans, forests], the predicted label is: mountains, with a confidence of 99.7%. If the candidate labels are: [Himalayas, Mount Tai, Mount Huang, Mount Hua], the predicted label is: Himalayas, with a confidence of 85.1%. Figure 4 (b) If the candidate labels are: [mountain, river, ocean, forest], the predicted label is: river, with a confidence of 92.6%. If the candidate labels are: [bridge, road, stairs, highway], the predicted label is: bridge, with a confidence of 99.9%. If the candidate labels are: [beam bridge, arch bridge, cable-stayed bridge, suspension bridge], the predicted label is: beam bridge, with a confidence of 79.4%. Figure 4 (c) If the candidate labels are: [mountain, river, ocean, forest], the predicted label is: ocean, with a confidence of 98.7%. If the candidate labels are: [oil well, airport, parking lot, basement], the predicted label is: oil well, with a confidence of 100.0%. Figure 4 (d) If the candidate labels are: [mountain, river, ocean, forest], the predicted label is: ocean, with a confidence level of 98.2%. If the candidate labels are: [oil well, airport, parking lot, basement], the predicted label is: oil well, with a confidence level of 100%.

[0063] From the above, we can see that the same geoscience document image contains different types of geoscience symbols, and the most accurate predicted label will be given according to the different candidate labels. Moreover, the same symbol will give the most accurate predicted label according to the accuracy of the candidate labels. For example Figure 4 In (b), when the candidate labels are [mountain, river, ocean, forest], the predicted label is: river. However Figure 4 The bridge in (b) is also a significant geoscientific feature. Therefore, when the candidate labels are determined as [bridge, road, stairs, highway], the predicted label is obtained: bridge. When the candidate labels are further refined to [beam bridge, arch bridge, cable-stayed bridge, suspension bridge], the predicted label is obtained: beam bridge.

[0064] This application realizes image understanding and classification in the field of geological academics, which can help to fully explore and conduct in-depth research on geological big data. The zero-sample and fine-tuning results of the Geo-CLIP multimodal pre-trained model on COCO-CN are shown in Table 1. It can be seen that the classification accuracy of the Geo-CLIP model in the zero-sample and fine-tuning experimental settings of the two tasks of text-image retrieval and image-text retrieval is significantly higher than that of the comparison model Wukong. The classification accuracy of the R2D2 model in the fine-tuning experimental setting is slightly higher than that of the Geo-CLIP model overall, but R2D2 does not support zero-sample image classification. The Geo-CLIP model proposed in the present invention achieved a relatively high classification accuracy in the zero-sample experimental setting of the two tasks, and the classification accuracy can be further improved after fine-tuning, indicating that the model has good generalization performance.

[0065] Table 1 Comparison between text-image retrieval and image-text retrieval

[0066]

[0067] Corresponding to the above embodiments, the present application also provides an embodiment of a zero-sample classification device for geological literature images.

[0068] See also Figure 5 , the zero-sample classification device 20 for geoscience literature images includes:

[0069] The parsing module 201 is used to parse out data pairs of geographical images and text descriptions from geographical documents, wherein the text descriptions are description sentences corresponding to the geographical images in the geographical documents.

[0070] The acquisition module 202 is used to obtain an image feature vector and a text feature vector according to the data pair.

[0071] The model training module 203 is used to obtain N positive sample pairs and N (N-1) negative sample pairs by using the image feature vector as a horizontal input vector and the text feature vector as a vertical input vector. The N positive sample pairs and N (N-1) negative sample pairs are used to train a pre-established Geo-CLIP model, where N is the number of geographic images.

[0072] The image classification module 204 is used to input the document image to be identified into the Geo-CLIP model, and the Geo-CLIP model generates a text description according to the document image to be identified, and the text description represents the classification result of the document image.

[0073] Corresponding to the above embodiment, an embodiment of the present application further provides an electronic device.

[0074] See also Figure 6, is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 6 As shown, the electronic device 300 may include: a processor 301, a memory 302 and a communication unit 303. These components communicate through one or more buses. Those skilled in the art can understand that the electronic device structure shown in the figure does not constitute a limitation on the embodiments of the present application. It can be a bus structure or a star structure, and can also include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0075] The communication unit 303 is used to establish a communication channel so that the electronic device can communicate with other devices.

[0076] The processor 301 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. It runs or executes software programs and / or modules stored in the memory 302, and calls data stored in the memory to perform various functions of the electronic device and / or process data. The processor can be composed of an integrated circuit (IC), for example, it can be composed of a single packaged IC, or it can be composed of multiple packaged ICs with the same or different functions. For example, the processor 301 can include only a central processing unit (CPU). In the embodiment of the present application, the CPU can be a single computing core or multiple computing cores.

[0077] The memory 302 is used to store the execution instructions of the processor 301. The memory 302 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0078] When the execution instructions in the memory 302 are executed by the processor 301 , the electronic device 300 is enabled to execute part or all of the steps in the above method embodiment.

[0079] Corresponding to the above embodiment, the embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium may store a program, wherein when the program is running, the device where the computer-readable storage medium is located may be controlled to perform some or all of the steps in the above method embodiment. In a specific implementation, the computer-readable storage medium may be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0080] Corresponding to the above-mentioned embodiment, the embodiment of the present application further provides a computer program product, which includes executable instructions. When the executable instructions are executed on a computer, the computer executes some or all of the steps in the above-mentioned method embodiment.

[0081] In the embodiments of the present application, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent the existence of A alone, the existence of A and B at the same time, and the existence of B alone. Among them, A and B can be singular or plural. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b and c can be represented by: a, b, c, ab, ac, bc, or abc, where a, b, c can be single or multiple.

[0082] Those of ordinary skill in the art will appreciate that the various units and algorithm steps described in the embodiments disclosed herein can be implemented in a combination of electronic hardware, computer software, and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0083] In several embodiments provided in the present application, any function can be stored in a computer-readable storage medium if it is implemented in the form of a software functional unit and sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., and other media that can store program codes.

[0084] The above is only a specific implementation of the present application. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. The protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A zero-sample classification method for geoscience literature images, characterized in that: include: Parsing data pairs of geographic images and text descriptions from geoscience documents, wherein the text descriptions are description sentences corresponding to the geographic images in the geoscience documents; Obtaining an image feature vector and a text feature vector according to the data pair; The image feature vector is used as a horizontal input vector, and the text feature vector is used as a vertical input vector to obtain N positive sample pairs and N (N-1) negative sample pairs, and the N positive sample pairs and N (N-1) negative sample pairs are used to train a pre-established Geo-CLIP model, where N is the number of geographic images; The document image to be identified is input into the Geo-CLIP model, and the Geo-CLIP model generates a text description according to the document image to be identified, wherein the text description represents a classification result of the document image.

2. The zero-sample classification method for geoscience literature images according to claim 1 is characterized in that: The data pairs of geographic images and text descriptions parsed from the geoscience literature include: Parsing the geography document by PDF to determine (geographical image, text description) pairs, wherein the text description in each (geographical image, text description) pair corresponds to the geographic image in the current pair; The geographic image and the text description in the (geographic image, text description) pair are separated.

3. The zero-sample classification method for geoscience literature images according to claim 2 is characterized in that: The Geo-CLIP model includes a picture editor and a text editor, and obtaining an image feature vector and a text feature vector according to the data pair includes: Inputting the geographic picture in the geography document into the picture editor and mapping it into the first feature space to obtain a first image feature vector; The text description in the geoscience document is input into the text editor and mapped to the second feature space to obtain a first text feature vector, and the number of the text feature vector is the same as that of the image feature vector.

4. The zero-sample classification method for geoscience literature images according to claim 3 is characterized in that: The feature vectors output by the image encoder and the text encoder have different lengths, and an additional linear mapping is used to map the first image feature vector and the first text feature vector to a uniform length.

5. The zero-sample classification method for geoscience literature images according to claim 4 is characterized in that: A loss function consisting of N positive sample pairs and N(N-1) negative sample pairs is used, and the Geo-CLIP model is then trained by contrastive learning.

6. The zero-sample classification method for geoscience literature images according to claim 1, characterized in that: The step of inputting the document image to be identified into the Geo-CLIP model, and the Geo-CLIP model generating a text description according to the document image to be identified, comprises: Mapping the document image to be identified through an image encoder to obtain a second image feature vector; Map the candidate category labels through the text encoder to obtain a text feature vector group; A second text feature vector in the text feature vector group having the greatest similarity to the second image feature vector is determined, and a text description of the second text feature vector is the classification result of the document image to be identified.

7. The zero-sample classification method for geoscience literature images according to claim 6 is characterized in that: Determining a second text feature vector in the text feature vector group that has the greatest similarity to the second picture feature vector includes: Comparing each text feature vector of the text feature vector group with the second image feature vector to obtain a confidence value; The text feature vector with the highest confidence value is used as the classified text feature vector of the second image feature vector.

8. A zero-sample classification device for geoscience literature images, characterized in that: include: A parsing module, used to parse out data pairs of geographical images and text descriptions from geographical documents, wherein the text descriptions are description sentences corresponding to the geographical images in the geographical documents; An acquisition module, used for acquiring an image feature vector and a text feature vector according to the data pair; A model training module, used to obtain N positive sample pairs and N (N-1) negative sample pairs by using the image feature vector as a horizontal input vector and the text feature vector as a vertical input vector, wherein the N positive sample pairs and the N (N-1) negative sample pairs are used to train a pre-established Geo-CLIP model, where N is the number of geographic images; The image classification module is used to input the document image to be identified into the Geo-CLIP model, and the Geo-CLIP model generates a text description based on the document image to be identified, and the text description represents the classification result of the document image.

9. An electronic device, characterized in that: include: processor; Memory; And a computer program, wherein the computer program is stored in the memory, and the computer program includes instructions, and when the instructions are executed by the processor, the electronic device executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Literature proofreading method and system, storage medium and equipment

    CN113887361A

  • Three-dimensional image classification method, apparatus and device, and computer readable storage medium

    CN114239560A

  • Cross-modal retrieval confrontation and defense method based on prompt learning

    CN115658954A

  • False news detection method based on multi-modal Transform

    CN115982350A