Zero-sample vector mosquito identification method and system

Abstract

The invention belongs to the technical field of image processing, and provides a zero-sample vector mosquito identification method and system, and the method comprises the steps: obtaining a to-be-identified mosquito image; preprocessing the acquired mosquito image, and extracting a local image of the mosquito image; constructing text description by a mosquito description generation method based on visual-semantic collaborative optimization; extracting local image region features and text description features of the local image and the text description respectively, calculating a local region image weight and a text description weight, and constructing a similarity matrix; and according to the local area image weight, the text description weight and the similarity matrix, calculating a weighted similarity score function, and according to the obtained weighted similarity score, identifying the type of the mosquito so as to complete the identification of the zero-sample vector mosquito.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and in particular relates to a zero-sample disease-carrying mosquito identification method and system. Background Art

[0002] The statements in this section merely provide background information related to the present invention and do not necessarily constitute prior art.

[0003] As mosquitoes that transmit a variety of infectious diseases (for example, Aedes aegypti and Aedes albopictus are the main vectors of dengue fever and Zika virus, and Anopheles sinensis is one of the main vectors of malaria), identifying their species can effectively control and accurately prevent and control infectious diseases. Therefore, identifying disease-transmitting mosquitoes has extremely important scientific research and application value.

[0004] Current mosquito identification methods primarily rely on microscopic observation by professionals, which is extremely inefficient and requires specialized expertise. With the advancement of machine vision, mosquito image recognition methods based on deep learning can improve mosquito identification efficiency. However, this requires large-scale labeled image datasets to train mosquito image recognition models, and high-quality, fine-grained mosquito image data is difficult to obtain. Mosquito images captured under natural conditions are characterized by diverse shooting angles, strong background interference, a wide variety of mosquito species, and subtle morphological differences, which exacerbates the difficulty of image annotation and model training. Samples of rare or unseen mosquito species are often scarce, resulting in poor generalization of traditional supervised learning-based methods in practical applications.

[0005] The image-text alignment capability based on a large visual language model can solve the problem that traditional machine vision requires large-scale labeled image datasets. Through cross-modal understanding capabilities, text descriptions are constructed to match mosquito images with text semantics without relying on large amounts of labeled data. It can demonstrate powerful zero-sample learning capabilities when identifying mosquitoes of unseen categories, and has overcome the problems that existing text descriptions are overly general and lack granularity and professionalism. Summary of the Invention

[0006] To solve the above problems, the present invention proposes a zero-sample disease-carrying mosquito identification method and system. By combining fine-grained text descriptions, the cross-modal associations between mosquito images and text descriptions are mined to complete zero-sample disease-carrying mosquito identification, thereby solving the problem that existing mosquito identification methods rely on a large number of manually annotated samples and the text descriptions are too general.

[0007] According to some embodiments, a first solution of the present invention provides a zero-sample disease-carrying mosquito identification method, which adopts the following technical solutions:

[0008] A zero-sample disease-carrying mosquito identification method, comprising:

[0009] Acquire a mosquito image to be identified;

[0010] Preprocessing the acquired mosquito image to extract a local image of the mosquito image;

[0011] Construct text descriptions based on a mosquito description generation method based on visual-semantic collaborative optimization;

[0012] Extract local image region features and text description features of local images and text descriptions respectively, calculate local region image weights and text description weights, and construct a similarity matrix;

[0013] Based on the local area image weight, text description weight and similarity matrix, a weighted similarity score function is calculated. The mosquito species are identified based on the obtained weighted similarity score, completing the zero-sample identification of disease-carrying mosquitoes.

[0014] As a further technical limitation, the process of constructing a text description by the mosquito description generation method based on visual-semantic collaborative optimization is as follows: guiding the language model to generate a structured description with visual discriminability according to a determined prompt template, and generating text for encapsulating the features and details of the category label based on the language model; that is, Among them, y∈Y, Y represents the category space, M represents the total number of generated text descriptions, h(y) represents the language model, y represents the category label, j represents the jth text description generated, y j Represents the j-th category label.

[0015] As a further technical limitation, the constructed similarity matrix S ij for in, Represents the local image region features, represents the text description feature, ||·|| represents the modulus of the feature vector (i.e., L2 norm), f(x i ) T represents f(x i )'s transposed matrix, cos(f(x i ),g(y j )) represents the eigenvector f(x i ) and the eigenvector g(y j ), d is the dimension of the feature vector, x i Represents the local image, y j Represents a text description.

[0016] Furthermore, the local area image weight w i It is used to measure the semantic consistency between the local area and the overall image, and the text description weight v j It is used to combine category-relevance filtering with category-independent descriptions and non-visual descriptions; i.e.

[0017]

[0018] Where M represents the total number of generated text descriptions, N represents the total number of local area image blocks obtained by image division, and k represents the index of the image block or text description currently participating in the calculation in the process (that is, the subscript variable when traversing all areas or all texts).

[0019] Furthermore, in the process of zero-shot disease-carrying mosquito identification, the mosquito image classification score is calculated according to the weighted similarity score function, and the mosquito species with the highest weighted similarity score is selected as the identification result to complete the zero-shot mosquito classification; that is, the weighted similarity score function S w for Recognition results for

[0020] As a further technical limitation, the process of extracting the local image of the mosquito image is: based on the preprocessed mosquito image and the adaptive region selection strategy, the preprocessed mosquito image is subjected to multi-scale random cropping, and the region containing the key features is retained to obtain the local image of the mosquito image.

[0021] According to some embodiments, a second solution of the present invention provides a zero-sample disease-carrying mosquito identification system, which adopts the following technical solutions:

[0022] A zero-sample disease-carrying mosquito identification system, comprising:

[0023] an acquisition module configured to acquire an image of a mosquito to be identified;

[0024] an extraction module configured to pre-process the acquired mosquito image and extract a partial image of the mosquito image;

[0025] A construction module configured to construct a text description based on a visual-semantic co-optimized mosquito description generation method;

[0026] A calculation module is configured to extract local image region features and text description features of the local image and text description respectively, calculate the local region image weight and text description weight, and construct a similarity matrix;

[0027] The recognition module is configured to calculate a weighted similarity score function based on the local area image weight, text description weight and similarity matrix, identify the mosquito species based on the obtained weighted similarity score, and complete the zero-sample identification of disease-carrying mosquitoes.

[0028] According to some embodiments, a third solution of the present invention provides a computer-readable storage medium, which adopts the following technical solution:

[0029] A computer-readable storage medium stores a program, which, when executed by a processor, implements the steps of the zero-sample disease-carrying mosquito identification method as described in the first embodiment of the present invention.

[0030] According to some embodiments, a fourth solution of the present invention provides an electronic device, which adopts the following technical solution:

[0031] An electronic device includes a memory, a processor, and a program stored in the memory and running on the processor. When the processor executes the program, the steps of the zero-sample disease-carrying mosquito identification method as described in the first embodiment of the present invention are implemented.

[0032] According to some embodiments, a fifth solution of the present invention provides a computer program product, which adopts the following technical solution:

[0033] A computer program product includes software code, wherein the program in the software code executes the steps of the zero-sample disease-carrying mosquito identification method as described in the first embodiment of the present invention.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] The present invention combines fine-grained text semantics with local image features, and can achieve mosquito identification without collecting training samples for each mosquito species, significantly reducing the cost of manual labeling and breaking away from the limitation of relying on large-scale labeled samples; it effectively identifies the tiny identification features of mosquitoes through local area alignment, and the weighted mechanism automatically filters irrelevant areas and low-quality descriptions, with low deployment cost. Based on the pre-trained model, it does not require additional training; it can identify rare or scarce mosquito species that are difficult to identify with traditional methods, is suitable for the identification of new or rare mosquito species, and has the ability of zero-sample learning. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The drawings constituting a part of the specification of this embodiment are used to provide a further understanding of this embodiment. The schematic embodiments and descriptions of this embodiment are used to explain this embodiment and do not constitute an improper limitation on this embodiment.

[0037] Figure 1 This is a flowchart of a zero-sample disease-carrying mosquito identification method in Example 1 of the present invention;

[0038] Figure 2 This is a step diagram of a zero-sample disease-carrying mosquito identification method in Example 1 of the present invention;

[0039] Figure 3 This is a schematic diagram of the structure of the visual language large model in the first embodiment of the present invention;

[0040] Figure 4This is a structural block diagram of a zero-sample disease-carrying mosquito identification system in Example 2 of the present invention. DETAILED DESCRIPTION

[0041] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0042] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention belongs.

[0043] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0044] In the present invention, terms such as "upper", "lower", "left", "right", "front", "back", "vertical", "horizontal", "side", "bottom", etc. indicating directions or positional relationships are based on the directions or positional relationships shown in the accompanying drawings. They are relational words determined only for the convenience of describing the structural relationships of the various parts or elements of the present invention, and do not specifically refer to any part or element in the present invention, and should not be understood as limiting the present invention.

[0045] In the present invention, terms such as "fixed connection," "connected," and "connection" should be interpreted broadly to mean a fixed connection, an integral connection, or a detachable connection; a direct connection or an indirect connection through an intermediary. Relevant researchers or technicians in this field may determine the specific meanings of these terms in the present invention based on specific circumstances, and they should not be construed as limitations of the present invention.

[0046] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0047] Example 1

[0048] Embodiment 1 of the present invention introduces a zero-sample disease-carrying mosquito identification method.

[0049] like Figure 1 and Figure 2 A zero-sample disease-carrying mosquito identification method is shown, comprising:

[0050] Acquire a mosquito image to be identified;

[0051] Preprocessing the acquired mosquito image to extract a local image of the mosquito image;

[0052] Construct text descriptions based on a mosquito description generation method based on visual-semantic collaborative optimization;

[0053] Extract local image region features and text description features of local images and text descriptions respectively, calculate local region image weights and text description weights, and construct a similarity matrix;

[0054] Based on the local area image weight, text description weight and similarity matrix, a weighted similarity score function is calculated. The mosquito species are identified based on the obtained weighted similarity score, completing the zero-sample identification of disease-carrying mosquitoes.

[0055] In this embodiment, mosquito images are collected by a camera or a monitoring terminal, and preprocessing operations such as image enhancement, denoising, cropping, and target detection are performed on the collected mosquito images. The lightweight YOLO-Mosquito model is used to locate the mosquito area in the mosquito image and output the bounding box (x min ,y min ,x max ,y max ), crop ROI image x roi ,Right now:

[0056] x roi =x(y min :y max ,x min :x max ,:),

[0057] Among them, x is the original image, and the last colon means that all channels (such as RGB channels) are retained. Let x roi is the new x.

[0058] In this embodiment, the process of constructing a text description based on the visual-semantic collaborative optimization mosquito description generation method is as follows: guiding the language model to generate a structured description with visual discriminability according to the determined prompt template, and generating text for encapsulating the features and details of the category label based on the language model; that is, Where y∈Y, Y represents the category space, M represents the total number of generated text descriptions, excluding non-visual descriptions; h(y) represents the language model, y represents the category label, j represents the jth text description generated, y j Represents the j-th category label.

[0059] There are five types of prompt templates in this embodiment, namely:

[0060] (1) "Describe what a(n){}looks like";

[0061] (2) "How can you identify a(n){}?";

[0062] (3) "What does a(n){}look like?";

[0063] (4) "A caption of an image of a(n){}";

[0064] (5) “Describe an image from the internet of a(n){}”.

[0065] It should be noted that the definition of visual relevance score Score(y j ), the specific formula is as follows:

[0066] Score(y j )=cos(g(y j ),t visual ),

[0067] Among them, t visual =g('morphological features') is the visual concept anchor, and descriptions with score ≥ 0.6 are retained.

[0068] As one or more implementation methods, the input mosquito image Perform multi-scale random cropping to obtain local image areas Adopting adaptive region selection strategy to retain the region containing key features, the specific cropping formula is As shown below:

[0069]

[0070] γ i :U(α,β),

[0071] Among them, α=0.4, β=0.9 control the size of the cropping area, W represents the width of the original image, H represents the height of the original image, and γ i represents a random scaling factor sampled from the interval [α, β], U represents a uniform distribution on the closed interval [α, β], and γ i is a random variable that follows this distribution, and N = 50 is the number of crops. Alternatively, a pre-trained segmentation model (such as SAM) can be used to locate the mosquito anatomical structure and crop the head, thorax, and other regions in a targeted manner.

[0072] The local image {x i} and text description {y j} are input into the pre-trained Figure 3The large visual language model shown in the figure consists of two independent encoders: the image encoder f(·) and the text encoder g(·). The visual encoder uses the Vision Transformer (ViT-B / 32), and the text encoder uses the Transformer structure to map images and text into the same semantic space. The training of the large visual language model is based on the contrastive learning objective, using large-scale image-text pairs (e.g., 400 million) to construct positive and negative samples between images and text. The image encoder f(·) and the text encoder g(·) are used to extract local image region features respectively. and text description features Construct the similarity matrix S between the local area and the text description ij ,Right now:

[0073]

[0074] Among them, the similarity matrix d is the dimension of the feature vector; ||·|| represents the modulus of the feature vector (i.e., L2 norm), f(x i ) T represents f(x i ), cos(f(x i ),g(y j )) represents the eigenvector f(x i ) and the eigenvector g(y j The visual language model structure used in the present invention is based on the CLIP public model and can also be fine-tuned through transfer learning combined with mosquito image and text data to adapt to specific field recognition tasks.

[0075] In this embodiment, a weight is assigned to each local region image and each text description, namely, the local region image weight w i , measures the semantic consistency between the local area and the overall image, and the text description weight v j Combine category-relevance filtering with category-independent descriptions and non-visual descriptions; that is:

[0076]

[0077] Where M represents the total number of generated text descriptions, N represents the total number of local area image blocks obtained by image division, and k represents the index of the image block or text description currently participating in the calculation in the process (that is, the subscript variable when traversing all areas or all texts).

[0078] In the process of zero-shot disease-carrying mosquito identification, the mosquito image classification score is calculated according to the weighted similarity score function, and the mosquito species with the highest weighted similarity score is selected as the identification result to complete the zero-shot mosquito classification; that is, the weighted similarity score function Sw for Recognition results for Based on the identification results, the mosquito species are associated with their corresponding disease transmission risks, and prevention and control recommendations are given in a timely manner.

[0079] This embodiment combines fine-grained text semantics with local image features, and can achieve mosquito identification without collecting training samples for each mosquito species, significantly reducing the cost of manual labeling and breaking away from the limitation of relying on large-scale labeled samples; it effectively identifies the tiny identification features of mosquitoes through local area alignment, and the weighted mechanism automatically filters out irrelevant areas and low-quality descriptions. It has low deployment costs and is based on pre-trained models, requiring no additional training; it can identify rare or scarce mosquito species that are difficult to identify with traditional methods, is suitable for the identification of new or rare mosquito species, and has the ability of zero-sample learning.

[0080] Example 2

[0081] The second embodiment of the present invention introduces a zero-sample disease-carrying mosquito identification system.

[0082] like Figure 4 A zero-sample disease-carrying mosquito identification system is shown, comprising:

[0083] an acquisition module configured to acquire an image of a mosquito to be identified;

[0084] an extraction module configured to pre-process the acquired mosquito image and extract a partial image of the mosquito image;

[0085] A construction module configured to construct a text description based on a visual-semantic co-optimized mosquito description generation method;

[0086] A calculation module is configured to extract local image region features and text description features of the local image and text description respectively, calculate the local region image weight and text description weight, and construct a similarity matrix;

[0087] The recognition module is configured to calculate a weighted similarity score function based on the local area image weight, text description weight and similarity matrix, identify the mosquito species based on the obtained weighted similarity score, and complete the zero-sample identification of disease-carrying mosquitoes.

[0088] The detailed steps are the same as those of the zero-sample disease-carrying mosquito identification method provided in Example 1 and will not be repeated here.

[0089] Example 3

[0090] A third embodiment of the present invention provides a computer-readable storage medium.

[0091] A computer-readable storage medium stores a program, which, when executed by a processor, implements the steps of the zero-sample disease-carrying mosquito identification method as described in the first embodiment of the present invention.

[0092] The detailed steps are the same as those of the zero-sample disease-carrying mosquito identification method provided in Example 1 and will not be repeated here.

[0093] Example 4

[0094] A fourth embodiment of the present invention provides an electronic device.

[0095] An electronic device includes a memory, a processor, and a program stored in the memory and running on the processor. When the processor executes the program, the steps of the zero-sample disease-carrying mosquito identification method as described in Example 1 of the present invention are implemented.

[0096] The detailed steps are the same as those of the zero-sample disease-carrying mosquito identification method provided in Example 1 and will not be repeated here.

[0097] Example 5

[0098] A fifth embodiment of the present invention provides a computer program product.

[0099] A computer program product includes software code, wherein the program in the software code executes the steps of the zero-sample disease-carrying mosquito identification method as described in the first embodiment of the present invention.

[0100] The detailed steps are the same as those of the zero-sample disease-carrying mosquito identification method provided in Example 1 and will not be repeated here.

[0101] It will be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present invention may be implemented in various computer languages, for example, the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0102] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0103] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0104] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0105] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0106] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

[0107] The above description is merely a preferred embodiment of this embodiment and is not intended to limit this embodiment. Those skilled in the art will readily appreciate that this embodiment may be modified and varied in various ways. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of this embodiment shall be within the scope of protection of this embodiment.

Claims

1. A zero-sample disease-carrying mosquito identification method, characterized in that: include: Acquire a mosquito image to be identified; Preprocessing the acquired mosquito image to extract a local image of the mosquito image; Construct text descriptions based on a mosquito description generation method based on visual-semantic collaborative optimization; Extract local image region features and text description features of local images and text descriptions respectively, calculate local region image weights and text description weights, and construct a similarity matrix; Based on the local area image weight, text description weight and similarity matrix, a weighted similarity score function is calculated. The mosquito species are identified based on the obtained weighted similarity score, completing the zero-sample identification of disease-carrying mosquitoes.

2. A zero-sample disease-carrying mosquito identification method as claimed in claim 1, characterized in that: The process of constructing text descriptions by the mosquito description generation method based on visual-semantic collaborative optimization is as follows: guiding the language model to generate a structured description with visual discriminability according to a determined prompt template, and generating text for encapsulating the features and details of the category label based on the language model; that is, Among them, y∈Y, Y represents the category space, M represents the total number of generated text descriptions, h(y) represents the language model, y represents the category label, j represents the jth text description generated, y j Represents the j-th category label.

3. A zero-sample disease-carrying mosquito identification method as claimed in claim 1, characterized in that: The constructed similarity matrix S ij for in, Represents the local image region features, represents the text description feature, ||·|| represents the modulus of the feature vector, f(x i ) T represents f(x i ), cos(f(x i ),g(y j )) represents the eigenvector f(x i ) and the eigenvector g(y j ), d is the dimension of the feature vector, x i Represents the local image, y j Represents a text description.

4. A zero-sample disease-carrying mosquito identification method as claimed in claim 3, characterized in that: The local area image weight w i It is used to measure the semantic consistency between the local area and the overall image, and the text description weight v j It is used to combine category-relevance filtering with category-independent descriptions and non-visual descriptions; i.e. Where M represents the total number of generated text descriptions, N represents the total number of local area image blocks obtained by image division, and k represents the index of the image block or text description currently participating in the calculation in the process.

5. A zero-sample disease-carrying mosquito identification method as claimed in claim 4, characterized in that: In the process of zero-shot disease-carrying mosquito identification, the mosquito image classification score is calculated according to the weighted similarity score function, and the mosquito species with the highest weighted similarity score is selected as the identification result to complete the zero-shot mosquito classification; that is, the weighted similarity score function S w for Recognition results for 6. A zero-sample disease-carrying mosquito identification method as claimed in claim 1, characterized in that: The process of extracting the local image of the mosquito image is as follows: based on the preprocessed mosquito image and the adaptive region selection strategy, the preprocessed mosquito image is subjected to multi-scale random cropping, and the region containing key features is retained to obtain the local image of the mosquito image.

7. A zero-sample disease-carrying mosquito identification system, characterized in that: include: an acquisition module, configured to acquire an image of a mosquito to be identified; an extraction module configured to pre-process the acquired mosquito image and extract a partial image of the mosquito image; A construction module configured to construct a text description based on a visual-semantic co-optimized mosquito description generation method; A calculation module is configured to extract local image region features and text description features of the local image and text description respectively, calculate the local region image weight and text description weight, and construct a similarity matrix; The recognition module is configured to calculate a weighted similarity score function based on the local area image weight, text description weight and similarity matrix, identify the mosquito species based on the obtained weighted similarity score, and complete the zero-sample identification of disease-carrying mosquitoes.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the zero-sample disease-carrying mosquito identification method according to any one of claims 1 to 6 are implemented.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the program, the steps of the zero-sample disease-carrying mosquito identification method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising software code, characterized in that The program in the software code executes the steps of the zero-sample disease-carrying mosquito identification method according to any one of claims 1 to 6.

Images