Image detection method, device, equipment, and computer program

Abstract

The present invention discloses an image detection method, device, equipment, and readable storage medium, which includes the steps of: performing a feature extraction process on an image to be detected to obtain at least two sub-image features; generating attention weights corresponding to each of the at least two sub-image features; performing a weighted aggregation process on the at least two sub-image features based on the attention weights to obtain a first feature vector; performing a clustering sampling process on the at least two sub-image features to obtain sampled sub-image features contained in each of the at least two classification clusters; determining block sparse self-attention corresponding to each sampled sub-image feature based on the at least two classification clusters and a block sparse matrix; determining a second feature vector based on the at least two block sparse self-attention; and determining a classification result of the image to be detected based on the first feature vector and the second feature vector. The present invention can improve the detection speed and detection accuracy of images.

Description

[Technical Field]

[0001] FIELD OF THE INVENTION This application relates to the field of computer technology, and more particularly to image detection technology.

[0002] This application claims priority to a Chinese patent application filed with the China Patent Office on March 23, 2022, bearing application number 2022102886990 and entitled "Image detection method, device, equipment, and readable storage medium," the entire contents of which are incorporated herein by reference. [Background technology]

[0003] In traditional machine learning image classification problems, it is often assumed that an image clearly belongs to a certain category. However, in real applications, multiple instances are observed in an image, and the image label only describes the category of one of the instances. This type of problem is generally called multiple instance learning (MIL). The goal of MIL is to create a multi-instance learning model by learning multiple instance images with classification labels, and then apply the multi-instance learning model to detect unknown multi-instance images.

[0004] In related art, a self-attention module mines information about all instances in an existing multi-instance image, finds related information between the instances, and creates a multi-instance learning model to detect unknown multi-instance images. However, the self-attention module has high computational complexity. When modeling multi-instance images, such as digitized pathology images, where the number of instances can reach approximately 10,000, the self-attention module consumes a lot of hardware resources and time, making training difficult. Furthermore, for multi-instance images, such as digitized pathology images, which are small in number and each image corresponds to only one overall image label, the supervision information is very weak. Therefore, when training a self-attention module with such high complexity on a small dataset, it is difficult to ensure that the self-attention module can mine effective information, and the self-attention module is prone to overfitting, resulting in low detection accuracy. Summary of the Invention [Means for solving the problem]

[0005] The embodiments of the present application provide an image detection method, device, apparatus, and readable storage medium, which can improve the image detection speed and detection accuracy.

[0006] In one aspect, the present embodiments provide an image detection method executed by a computing device, the method comprising: acquiring an image of the target to be detected, and performing a feature extraction process on the image of the target to obtain a feature representation subset of the image of the target to be detected, wherein the image of the target to be detected includes at least two sub-images, and the feature representation subset includes at least two sub-image features, and the at least two sub-image features correspond one-to-one to the at least two sub-images; generating attention weights corresponding to the at least two sub-image features, and performing a weighted aggregation process on the at least two sub-image features based on the attention weights to obtain a first feature vector; performing a clustering sampling process on at least two sub-image features to obtain at least two classification clusters, each of which includes the sampled sub-image features; determining block sparse self-attentions corresponding to each sampled sub-image feature based on the at least two classification clusters and the block sparse matrix; and determining a second feature vector based on the at least two block sparse self-attentions, wherein the block sparse self-attentions corresponding to the sampled sub-image features are determined based on the sampled sub-image features in the classification cluster to which the sampled sub-image features belong; and determining a classification result for the image to be detected based on the first feature vector and the second feature vector.

[0007] In one aspect, the present embodiments provide an image detection method executed by a computing device, the method comprising: obtaining a sample image and performing a feature extraction process on the sample image to obtain a sample feature representation subset of the sample image, wherein the sample image includes at least two sample sub-images, and the sample feature representation subset includes at least two sample sub-image features, and the at least two sample sub-image features have a one-to-one correspondence with the at least two sample sub-images; inputting at least two sample sub-images into an initial image discrimination model, generating sample attention weights corresponding to each of the at least two sample sub-image features through the initial image discrimination model, and performing a weighted aggregation process on the at least two sample sub-image features based on the sample attention weights corresponding to each of the at least two sample sub-image features to obtain a first sample feature vector; performing a clustering sampling process on at least two sample sub-image features using an initial image discrimination model to obtain at least two sample classification clusters, each of which includes the sample sampling sub-image features; determining sample block sparse self-attentions corresponding to each sample sampling sub-image feature based on the at least two sample classification clusters and the block sparse matrix; and determining a second sample feature vector based on the at least two sample block sparse self-attentions, wherein the sample block sparse self-attentions corresponding to each sample sampling sub-image feature are determined based on the sample sampling sub-image features in the sample classification cluster to which the sample sampling sub-image feature belongs; determining a sample classification result of the sample image based on the first sample feature vector and the second sample feature vector according to an initial image discrimination model; The method includes a step of adjusting model parameters for the initial image identification model based on at least two sample classification clusters, attention weights corresponding to each of the at least two sample sub-image features, the sample classification results, and the classification labels corresponding to the sample images, to obtain an image identification model that can be used to identify the classification results of the image to be detected.

[0008] In one aspect, the present embodiments provide an image detection device, comprising: a feature extraction module for acquiring an image of a target object, performing a feature extraction process on the image of the target object, and obtaining a feature representation subset of the image of the target object, wherein the image of the target object includes at least two sub-images, and the feature representation subset includes at least two sub-image features, and the at least two sub-image features correspond one-to-one with the at least two sub-images; a first vector generation module used to generate attention weights corresponding to at least two sub-image features, and perform weighted aggregation processing on the at least two sub-image features based on the attention weights to obtain a first feature vector; a second vector generation module for performing a clustering sampling process on at least two sub-image features to obtain at least two classification clusters, each of which includes the sampled sub-image features; determining block sparse self-attentions corresponding to each sampled sub-image feature based on the at least two classification clusters and the block sparse matrix; and determining a second feature vector based on the at least two block sparse self-attentions, where the block sparse self-attentions corresponding to the sampled sub-image features are determined based on the sampled sub-image features in the classification cluster to which the sampled sub-image features belong; A classification module is used to determine a classification result of the image to be detected based on the first feature vector and the second feature vector.

[0009] In one aspect, the present embodiments provide an image detection device, comprising: a sample feature extraction module used to acquire a sample image, perform feature extraction processing on the sample image, and obtain a sample feature representation subset of the sample image, wherein the sample image includes at least two sample sub-images, and the sample feature representation subset includes at least two sample sub-image features, and the at least two sample sub-image features have a one-to-one correspondence with the at least two sample sub-images; a first sample vector generation module used to input at least two sample sub-images into an initial image identification model, generate sample attention weights corresponding to each of the at least two sample sub-image features using the initial image identification model, and perform a weighted aggregation process on the at least two sample sub-image features based on the sample attention weights corresponding to each of the at least two sample sub-image features to obtain a first sample feature vector; a second sample vector generation module for performing a clustering sampling process on at least two sample sub-image features using the initial image discrimination model to obtain at least two sample classification clusters, each of which includes the sample sampling sub-image features; determining sample block sparse self-attentions corresponding to each sample sampling sub-image feature based on the at least two sample classification clusters and the block sparse matrix; and determining a second sample feature vector based on the at least two sample block sparse self-attentions, where the sample block sparse self-attentions corresponding to the sample sampling sub-image features are determined based on the sample sampling sub-image features in the sample classification cluster to which they belong; a sample classification module used for determining a sample classification result of the sample image according to the first sample feature vector and the second sample feature vector according to the initial image identification model; The training module is used to adjust model parameters of the initial image identification model based on at least two sample classification clusters, attention weights corresponding to each of the at least two sample sub-image features, the sample classification results, and the classification labels corresponding to the sample images, to obtain an image identification model used to identify the classification results of the image to be detected.

[0010] In one aspect, embodiments of the present application provide a computer device including a processor, a memory, and a network interface; The processor is connected to the memory and the network interface, where the network interface is used to provide a data communication network element, the memory is used to store a computer program, and the processor is used to call the computer program and thereby perform the method in the present embodiment.

[0011] In one aspect, the present embodiment provides a computer-readable storage medium having a computer program stored therein, the computer program being suitable for being loaded by a processor and for performing a method in the present embodiment.

[0012] In one aspect, the present embodiment provides a computer program product or a computer program including computer instructions stored in a computer-readable storage medium, the computer instructions being read by a processor of a computing device from the computer-readable storage medium, and the processor executing the computer instructions to cause the computing device to perform a method in the present embodiment.

[0013] In the present embodiment, a feature extraction process is performed on a target image including at least two sub-images to obtain a feature representation subset of the target image, the feature representation subset including sub-image features corresponding to each of the at least two sub-images. Then, information on the sub-images is mined using two methods: the first is to independently mine information on each sub-image, i.e., generate attention weights corresponding to each of the at least two sub-image features, and perform a weighted aggregation process on the at least two sub-image features based on the attention weights to obtain a first feature vector; the second is to mine related information between sub-images of the same category, i.e., perform a clustering sampling process on the at least two sub-image features to obtain sampled sub-image features included in each of the at least two classification clusters, determine block sparse self-attention corresponding to each sampled sub-image feature based on the at least two classification clusters and a block sparse matrix, determine a second feature vector based on the at least two block sparse self-attentions, and finally determine a classification result for the target image based on the first feature vector and the second feature vector. By adopting the method provided in the embodiments of the present application, the first and second feature vectors obtained by the two information mining methods can complement and constrain each other, thereby improving the accuracy of image detection. Furthermore, by calculating the block sparse self-attention corresponding to the sampled sub-image features using a block sparse matrix, it is possible to ensure that only the relationship between the sampled sub-image features and the sampled sub-image features belonging to the common classification cluster is paid attention to, thereby reducing the computational complexity and improving the detection speed. [Brief explanation of the drawings]

[0014] [Figure 1a] FIG. 1 is a schematic diagram of a network architecture provided by an embodiment of the present application. [Figure 1b] 1 is a schematic diagram illustrating an application scenario of the image detection method provided in the present embodiment; [Figure 2]FIG. 1 is a flow diagram of an image detection method provided in an embodiment of the present application. [Figure 3] 1 is a scene diagram of an image feature extraction process provided by an embodiment of the present invention; [Figure 4] FIG. 1 is a flow diagram of an image detection method provided in an embodiment of the present application. [Figure 5] FIG. 1 is a schematic diagram of the clustering results of colorectal pathology images provided in the present example. [Figure 6] FIG. 1 is a schematic diagram illustrating the principle of applying block sparsity constraints to global self-attention provided by an embodiment of the present application. [Figure 7] FIG. 1 is a structural schematic diagram of an image recognition model provided in an embodiment of the present application. [Figure 8] 1 is a flow diagram of an initial image recognition model training method provided in an embodiment of the present application; [Figure 9] 1 is a structural schematic diagram of an image detection device provided in an embodiment of the present application; [Figure 10] 1 is a structural schematic diagram of a computer device provided by an embodiment of the present application; [Figure 11] FIG. 2 is a structural schematic diagram of another image detection device provided by an embodiment of the present application; [Figure 12] FIG. 2 is a structural schematic diagram of another computer device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE INVENTION

[0015] The following describes the technical solutions in the embodiments of the present application clearly and completely, in combination with the drawings in the embodiments of the present application, and it is obvious that the described embodiments are only some of the embodiments of the present application, and not all of the embodiments, and all other embodiments that a person skilled in the art can obtain based on the embodiments of the present application without requiring creative efforts also fall within the scope of protection of the present application.

[0016] The solutions provided by the embodiments of the present application relate to artificial intelligence computer vision technology, machine learning, deep learning, and other technologies, which are specifically described in the following embodiments.

[0017] 1a, which is a schematic diagram of a network architecture provided by an embodiment of the present application. As shown in FIG. 1a, the network architecture may include a service server 100 and a group of terminal devices, which may include terminal device 10a, terminal device 10b, terminal device 10c, ..., and terminal device 10n, where any of the terminal devices in the group of terminal devices may have a communication connection with the service server 100. For example, there is a communication connection between terminal device 10a and the service server 100, a communication connection between terminal device 10b and the service server 100, and a communication connection between terminal device 10c and the service server 100. The communication connections are not limited to any particular method, and may be directly or indirectly connected via wired communication, directly or indirectly connected via wireless communication, or may be connected via other methods, and are not limited to these methods in the present application.

[0018] It should be understood that each of the terminal devices in the terminal group shown in FIG. 1A may have an application client terminal installed therein. When the application client terminal is running on each terminal device, it can exchange data with the service server 100 shown in FIG. 1A, thereby allowing the service server 100 to receive service data from each terminal device. Here, the application client terminal may be an application client terminal having a game application, a video editing application, a social application, an instant communication application, a live streaming application, a short video application, a video application, a music application, a shopping application, a novel application, a payment application, or a browser or other related image processing function. Here, the application client terminal may be an independent client terminal or an embedded sub-client terminal integrated in a client terminal (e.g., an instant communication client terminal, a social client terminal, a video client terminal, etc.), and is not limited thereto.

[0019] 1a, each terminal device in the terminal device group can operate the application client terminal to obtain a detection target image and send it as service data to the service server 100. The service server 100 can perform image detection on the detection target image and thereby determine a classification result for the detection target image. Here, the detection target image may also be referred to as a multi-instance image, i.e., it includes at least two sub-images, and one sub-image may be referred to as one instance. For a multi-instance image, if one instance is abnormal, the multi-instance image may be considered an abnormal image. In other words, if one sub-image is abnormal, the classification result of the detection target image should be an abnormal image.

[0020] In one possible embodiment, the image to be detected is a digitized pathology image. The digitized pathology image can be obtained by using an automated microscope or optical magnification system to scan a carrier slice to collect a high-resolution digital image. A computer then automatically performs high-precision multi-view seamless splicing and processing on the obtained high-resolution digital image, thereby obtaining high-quality visualization data, i.e., a digitized pathology image. The digitized pathology image can be zoomed in and out at any position in a computer device, and does not suffer from image distortion or unclear details. This makes it more convenient for doctors to perform pathological diagnoses such as cancer diagnosis, survival stage detection, and gene mutation detection compared to observing the original carrier slice. However, because digitized pathology images have high resolution, are often very large in size, and contain a large number of instances (biological tissues such as cells and genes), manually observing the digitized pathology images often requires continuous adjustment of the detection position and detection magnification, which requires a large amount of time and effort. Therefore, the digitized pathology image is loaded into the application client terminal, and after the terminal device acquires the digitized pathology image, it can be transmitted to the service server 100 as service data. Furthermore, the service server 100 can perform image detection on the digitized pathology image and determine a classification result for the digitized pathology image. The classification result can assist doctors in making medical diagnoses.

[0021] After acquiring the target image, the service server 100 performs image detection on the target image and determines a classification result for the target image. For a specific implementation process, please refer to FIG. 1b, which is a schematic diagram of an application scenario of the image detection method provided in the present embodiment. For ease of understanding, the target image will still be described as a digitized pathology image in the above embodiment. As shown in FIG. 1b, a patient management application 300 is installed on a terminal device 200 (which may be any of the terminal devices in FIG. 1a, for example, terminal device 10a), and a subject A has an association relationship with the terminal device 200. Assuming that subject A is the attending physician of subject B, subject A can view subject B's case file, for example, a colorectal pathology image 301, on the patient management application 300. By viewing the colorectal pathology image 301, subject A can diagnose whether subject B has colorectal cancer. Because the image size of the colorectal pathology image 301 is very large and there are many cellular tissues that need to be observed, it takes a long time for subject A to manually observe it. Therefore, subject A can initiate an image detection request for the colorectal pathology image 301 to a service server 400 (e.g., the service server 100 shown in FIG. 1a above) through a patient management application 300 running on the terminal device 200. The service server 400 then performs image detection on the colorectal pathology image 301 and determines a classification result for the colorectal pathology image 301, i.e., whether the colorectal pathology image 301 is a normal image or an abnormal image. The classification result of the colorectal pathology image 301 by the service server 400 can assist subject A in diagnosing the condition of subject B.

[0022] As can be understood, the colorectal pathology image 301 has a large image size and contains a large amount of cellular tissue, and therefore can be considered to contain at least two sub-images in the colorectal pathology image 301 (i.e., at least two sub-images can be obtained by dividing the colorectal pathology image 301), and if an abnormality is present in one sub-image, the colorectal pathology image 301 is an abnormal image.

[0023] As shown in FIG. 1b, after the terminal device 200 transmits the colorectal pathology image 301 to the service server 400, the service server 400 first performs a feature extraction process on the colorectal pathology image 301 to obtain a feature representation subset 401 used to represent the colorectal pathology image 301, where the feature representation subset 401 includes at least two sub-image features, and each sub-image feature is used to describe the information of one sub-image in the colorectal pathology image 301. Then, the service server 400 can detect the colorectal pathology image 301 using an image identification model 402, which may include a first attention sub-network 4021, a second attention sub-network 4022, and a classification sub-network 4023, where the first attention sub-network 4021 is used to regard sub-images as independent instances, mine independent representation information of each sub-image based on its respective sub-image features, and obtain a first feature vector used to represent the independent representation information; the second attention sub-network 4022 is used to mine global representation information among all sub-images, and obtain a second feature vector used to represent the global representation information; and the classification sub-network 4023 is used to perform image classification on the target image based on the first feature vector and the second feature vector, and determine the classification result of the target image.

[0024] As shown in FIG. 1b, after the service server 400 inputs the feature representation subset 401 into the image identification model 402, the first attention sub-network 4021 generates attention weights corresponding to each of the at least two sub-image features, and then performs a weighted aggregation process on the at least two sub-image features based on the attention weights to obtain a first feature vector 403. At the same time, the second attention sub-network 4022 performs a clustering sampling process on the at least two sub-image features to obtain sampled sub-image features included in each of the at least two classification clusters, and then determines block sparse self-attentions corresponding to each sampled sub-image feature based on the at least two classification clusters and the block sparse matrix, and determines a second feature vector 404 based on the at least two block sparse self-attentions. After determining the first feature vector 403 and the second feature vector 404, the classification subnetwork 4023 can perform a feature fusion process on the first feature vector 403 and the second feature vector 404 to obtain a fused feature vector. Then, the classification subnetwork 4023 performs a classification process on the fused feature vector to obtain a classification result 405 of the colorectal pathology image 301. The classification result 405 can include a normal probability and an abnormal probability of the colorectal pathology image 301. The normal probability refers to the probability that the colorectal pathology image 301 is a normal image, i.e., the probability that subject B is not affected by disease, and the abnormal probability refers to the probability that the colorectal image 301 is an abnormal image, i.e., the probability that subject B may be affected by colorectal cancer. The service server 400 will return the classification result 405 to the terminal device 200, and subject A can diagnose the disease status of subject B based on the classification result 405.

[0025] Optionally, if the image identification model 402 is stored locally in the terminal device 200, the terminal device 200 can locally perform an image detection task on the image to be detected and obtain a classification result for the image to be detected. Because the training image identification model 402 involves a large amount of offline calculation, the local image identification model of the terminal device 200 may be trained by the service server 400 and then transmitted to the terminal device.

[0026] As can be understood, the method provided by the embodiments of the present application may be performed by a computer device, including, but not limited to, a terminal device or a server. The service server 100 in the embodiments of the present application may be a computer device, and the terminal devices in a terminal device group may also be computer devices, but are not limited thereto. The server may be an independent physical server, a server group consisting of multiple physical servers, or a distributed system. It may also be a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDNs, big data, and artificial intelligence platforms. The terminal devices include, but are not limited to, mobile phones, computers, smart voice interaction devices, smart home appliances, in-vehicle terminals, etc. The embodiments of the present application may be applied to various scenarios, including, but not limited to, cloud technology, cloud security, domain blockchain, artificial intelligence, smart transportation, and driver assistance.

[0027] As can be understood, when the embodiments of the present application are applied to a specific product or technology, relevant data such as images of the relevant detected object must be obtained after obtaining user permission or consent, and the collection, use, and processing of the relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0028] It is necessary to note that in the present embodiment, the image to be detected is described as a colorectal pathology image as an example, but in actual application scenarios, the image to be detected may also be a pathology image of other types of cancer, or other multi-instance images containing at least two sub-images, and the present application does not limit it here.

[0029] Further, referring to FIG. 2, FIG. 2 is a flow diagram of an image detection method provided by an embodiment of the present application. Here, the image detection method may be executed by a computer device, where the computer device may be the service server 100 shown in FIG. 1a above, or any terminal device in the terminal device group shown in FIG. 1a above, for example, terminal device 10c. Hereinafter, an example in which the image detection method is executed by a computer device will be described. Here, the image detection method may include at least the following steps S101 to S104.

[0030] Step S101: Obtain an image of the detection target, perform a feature extraction process on the image of the detection target, and obtain a feature representation subset of the image of the detection target, where the image of the detection target includes at least two sub-images, the feature representation subset includes at least two sub-image features, and the at least two sub-image features correspond one-to-one to the at least two sub-images.

[0031] Specifically, the image to be detected is a weak image annotation, a multi-instance image in which multiple instances correspond to one label. A multi-instance image, also called a multi-instance bag, contains several instances, and an instance may be considered as a sub-image, but only the bag contains a label, and the instances do not contain labels. If a multi-instance bag contains at least one positive instance, the bag is marked as a positive-class multi-instance bag (positive bag). If all instances in the multi-instance bag are negative instances, the bag is marked as a negative-class multi-instance bag (negative bag). The image to be detected may be a digitized pathology image applied in pathological diagnosis such as cancer diagnosis, survival prediction, gene mutation prediction, etc., and image detection is performed on the digitized pathology image to obtain a classification result for the digitized pathology image, which can assist doctors in determining the corresponding medical diagnosis result. For example, the digitized pathology image is the colorectal pathology image 301 in FIG. 1b above. Image detection is performed on the colorectal pathology image 301 to obtain a classification result 405, which can assist subject A in determining whether subject B is suffering from colorectal cancer.

[0032] Specifically, the image to be detected is media data used by humans and lacks information that can be understood by computers. Therefore, the image to be detected needs to be converted from an unstructured original image into structured information that can be identified and processed by computers. That is, it is necessary to perform scientific abstraction on the image to be detected, create a mathematical model of it, and use it to describe and replace the image to be detected, and have a computer realize the identification of the image to be detected through calculations and operations on the mathematical model. The mathematical model may be a vector space model, and in this case, sub-image features corresponding to sub-images contained in the image to be detected may be vectors in the vector space model, and the computer can describe and apply the image to be detected using a feature representation subset composed of the sub-image features.

[0033] Specifically, if all sub-image features are treated as feature items, it will result in a large amount of calculation. Therefore, it is necessary to simplify the calculation and improve the speed and efficiency of the image processing by reducing the number of sub-images to be processed as much as possible without damaging the core information of the image to be detected. Therefore, one possible specific process for performing feature extraction on the image to be detected and obtaining a feature representation subset of the image to be detected is as follows: identify the background region and the foreground region in the image to be detected, then perform image segmentation on the image to be detected based on the background region and the foreground region to obtain the foreground image to be detected, then perform a scaling process on the foreground image to be detected based on a scaling factor to obtain the foreground image to be cropped, and then perform a cropping process on the foreground image to be cropped based on the predetermined length and the predetermined width of the sub-image to obtain at least two sub-images, finally perform image feature extraction on each of the at least two sub-images to obtain sub-image features corresponding to each of the at least two sub-images, and determine a feature representation subset of the image to be detected based on the sub-image features corresponding to each of the at least two sub-images. Here, the preset length of the sub-image is less than the length of the foreground image to be trimmed, and the preset width of the sub-image is less than the width of the foreground image to be trimmed.

[0034] To easily understand the specific steps of the feature extraction process, please refer to FIG. 3, which is a scene diagram of the image feature extraction process provided by the embodiment of the present application. As shown in FIG. 3, after acquiring a target image 3000, a computer device first performs foreground / background classification on the target image 3000 to determine a foreground region 3001 and a background region 3002 contained in the target image 3000. Then, image segmentation of the foreground / background region is performed to obtain a target foreground image 3003. Next, a cropping process is performed on the target foreground image 3003 based on a predetermined sub-image length and a predetermined sub-image width, e.g., 512*512, to obtain at least two sub-images, i.e., sub-image 3004, sub-image 3005, ..., and sub-image 3006. Next, the at least two sub-images are input to a feature extractor 3007, which extracts sub-image features corresponding to each sub-image, thereby obtaining a feature representation subset 3008. Here, the feature extractor 3007 may be implemented using a feature extractor RestNet50 (Residual net 50) or other pre-trained networks, and this application is not limited thereto. The foreground image 3003 to be detected can be selectively enlarged or reduced at a set magnification. For example, enlarging the foreground image 3003 to be detected by ten times to obtain a foreground image to be cropped, and then cropping the foreground image to be cropped, can obtain more sub-images than when cropping the foreground image 3003 to be detected. Therefore, more sub-image features can be obtained, and the image to be detected can be expressed more precisely.

[0035] Step S102: Generate an attention weight corresponding to each of the at least two sub-image features, and perform a weighted aggregation process on the at least two sub-image features according to the attention weight to obtain a first feature vector.

[0036] Specifically, the attention weight, also referred to as the attention score, is used to measure the importance of a sub-image feature, and the larger the attention weight, the higher the importance of the corresponding sub-image feature, and the greater the proportion of the corresponding sub-image feature in the final output first feature vector. The attention weight corresponding to each sub-image feature can be obtained by a single network learning using the sub-image feature itself as input.

[0037] Specifically, after obtaining the attention weights, a weighted aggregation process can be performed on the sub-image features, that is, a weighted sum is calculated for the sub-image features based on the attention weights to obtain a first feature vector.

[0038] Step S103: Perform a clustering sampling process on the at least two sub-image features to obtain at least two classification clusters, where the classification clusters include the sampled sub-image features; determine block sparse self-attentions corresponding to each of the sampled sub-image features based on the at least two classification clusters and a block sparse matrix; determine a second feature vector based on the at least two block sparse self-attentions, where the block sparse self-attentions corresponding to the sampled sub-image features are determined based on the sampled sub-image features in the classification cluster to which they belong.

[0039] Specifically, in order to better mine the associations between each sub-image feature and avoid excessive computational complexity, at least two sub-image features can be clustered, i.e., at least two sub-image features can be divided into at least two classification clusters based on the similarity of the sub-image features, and sub-images corresponding to the sub-image features in one classification cluster belong to the same category of images. Then, some sub-image features from each classification cluster are sampled as sampled sub-image features.

[0040] Specifically, the self-attention of the sampled subimage features is determined based on a global self-attention weight matrix, where the global self-attention weight matrix is ​​used to characterize the degree of association between the sampled subimage features. Because the sampled subimage features have already been classified, when determining the self-attention of a sampled subimage feature, the computer device only needs to pay attention to sampled subimage features that belong to the same classification cluster as the sampled subimage feature. After determining the global self-attention weight matrix based on the sampled subimage features, the computer device can obtain matching block sparse matrices for at least two classification clusters, and filter the global self-attention weight matrix based on the block sparse matrix to obtain a block sparse global self-attention weight matrix, which is used to characterize the degree of association between the sampled subimage features in the same classification cluster. Then, the computer device can determine the block sparse self-attention of each sampled subimage feature based on the block sparse global self-attention weight matrix. The computer device then performs an average pooling process on the block sparse self-attentions of all the sampled subimage features to obtain a second feature vector.

[0041] Step S104: Based on the first feature vector and the second feature vector, a classification result of the detection target image is determined.

[0042] Specifically, after obtaining the first feature vector and the second feature vector, an MLP (Multilayer Perceptron) classifier can be adopted to predict the first feature vector and the second feature vector and output the classification result.

[0043] The method provided in the present embodiment employs a feature extraction process for a target image including at least two sub-images to obtain a feature representation subset for the target image, the feature representation subset including sub-image features corresponding to each of the at least two sub-images. Then, information about the sub-images is mined using two methods: first, information about each sub-image is independently mined, i.e., attention weights corresponding to each of the at least two sub-image features are generated, and a weighted aggregation process is performed on the at least two sub-image features based on the attention weights to obtain a first feature vector; second, related information between sub-images of the same category is mined, i.e., a clustering sampling process is performed on the at least two sub-image features to obtain at least two classification clusters, each of which includes sampled sub-image features; block sparse self-attention corresponding to each sampled sub-image feature is determined based on the at least two classification clusters and a block sparse matrix; a second feature vector is determined based on the at least two block sparse self-attention; and finally, a classification result for the target image is determined based on the first feature vector and the second feature vector. The first and second feature vectors obtained by the two information mining methods can complement and constrain each other, thereby improving the accuracy of image detection. Furthermore, when calculating the block sparse self-attention corresponding to the sampled sub-image features using the block sparse matrix, it is possible to ensure that only the relationship between the sampled sub-image features and the sampled sub-image features belonging to the common classification cluster is paid attention to, thereby reducing the computational complexity and improving the detection speed.

[0044] Further, as shown in FIG. 4, FIG. 4 is a flow diagram of an image detection method provided by an embodiment of the present application. Here, the image detection method may be executed by a computer device, and the computer device may be the service server 100 shown in FIG. 1a above, or any terminal device in the terminal device group shown in FIG. 1a above, for example, terminal device 10c. Hereinafter, the image detection method will be described taking the example of being executed by a computer device. Here, the image detection method may include at least the following steps S201 to S210.

[0045] Step S201: Obtain an image of the detection target, perform a feature extraction process on the image of the detection target, and obtain a feature representation subset of the image of the detection target, where the image of the detection target includes at least two sub-images, the feature representation subset includes at least two sub-image features, and the at least two sub-image features correspond one-to-one to the at least two sub-images.

[0046] Specifically, for the implementation of step S201, reference can be made to the specific description of step S101 in the embodiment corresponding to FIG. 2 above, and detailed description thereof will be omitted here.

[0047] Specifically, suppose the image to be detected is X. After the tissue-background segmentation and image cropping, at least two sub-images are obtained. The image set consisting of at least two sub-images is {x1, x2, ..., x n}, where each sub-image x i is called an instance of an image to be detected. The above process of performing feature extraction processing on at least two sub-images can be expressed by the following equation (1).

number

number

[0048] Step S202: Input the at least two sub-image features into a first attention sub-network in an image discrimination model, where the first attention sub-network includes a weight learning network layer and a weight summation network layer.

[0049] Step S203: The weight learning network layer performs a weight fitting process on the at least two sub-image features respectively to obtain attention weights corresponding to the at least two sub-image features.

[0050] Specifically, the weight learning network layer employs a parameterized neural network to learn attention weights corresponding to sub-image features, where the sub-image features h in the feature representation subset H obtained from Equation (1) above are k The corresponding attention weight α k may be expressed as equation (2).

number

[0051] Step S204: The weighted aggregation network layer performs a weighting process on each sub-image feature based on the attention weight to obtain a weighted sub-image feature corresponding to each of the sub-image features, and performs an aggregation process on the weighted sub-image features corresponding to each of the at least two sub-image features to obtain a first feature vector.

[0052] Specifically, in the weighted aggregation network layer, the sub-image features can be effectively aggregated by sampling the nonlinear attention weights of the linear terms, that is, calculated by Equation (3).

number

[0053] Step S205: Input the at least two sub-image features into a second attention sub-network in the image identification model, where the second attention sub-network includes a clustering sampling network layer, a global self-attention network layer, a self-attention network layer, and an average pooling network layer.

[0054] Specifically, inputting at least two sub-image features into the first attention sub-network and inputting at least two sub-image features into the second attention sub-network can be performed simultaneously, and the first attention sub-network and the second attention sub-network do not affect each other.

[0055] Step S206: The clustering sampling network layer performs a clustering sampling process on the at least two sub-image features to obtain at least two classification clusters, each of which includes sampled sub-image features, and the sum of the quantity of the sampled sub-image features included in each of the at least two classification clusters is N, where N is a positive integer less than the quantity of the at least two sub-image features.

[0056] Specifically, in the clustering sampling network layer, the computer device can first perform a clustering process on at least two sub-image features to obtain at least two classification clusters, then obtain a k-th classification cluster in the at least two classification clusters, where k is a positive integer, and the k-th classification cluster includes at least one clustering sub-image feature, then the computer device can obtain a vector distance between each of the at least one clustering sub-image feature and the cluster center of the k-th classification cluster as a reference distance, and sequentially obtain h clustering sub-image features in the at least one clustering sub-image feature based on the reference distance, and the h clustering sub-image features are the sampling sub-image features included in the k-th classification cluster, where h is a positive integer and h is less than or equal to the quantity of the at least one clustering sub-image feature.

[0057] As can be understood, assuming that the number of classification clusters obtained by performing clustering processing on at least two sub-image features is p, each classification cluster has one cluster center, and when sampling each classification cluster, h sub-image features that are closest to the cluster center of the classification cluster can be collected, and finally, a total of N sampled sub-image features can be obtained, where N=p×h. To reduce the complexity of calculation, the default sampling total N=128. After the clustering processing and sampling processing, the obtained N sampled sub-image features have diversity and can be approximated as an effective representative of the original at least two sub-image features.

[0058] Specifically, the clustering process may employ unsupervised k-means clustering (unsupervised clustering) or other clustering methods, without limitation. In at least two classification clusters obtained after clustering, the sub-images corresponding to the sub-image features contained in each classification cluster belong to the same image category. For ease of understanding, please also refer to FIG. 5, which is a schematic diagram of the clustering results of colorectal pathology images provided by the present embodiment. As shown in FIG. 5, sub-image features corresponding to sub-images in a common group belong to a common classification cluster. As can be seen from FIG. 5, different pathology images can be classified into categories with similar approximate histological structural features. For example, the sub-images in group 1 contain significant cancer tissue, the sub-images in group 2 all have abnormal staining or relatively poor imaging quality, the sub-images in group 3 are primarily glandular tissue, and the sub-images in group 4 all contain a relatively large number of immune cells, etc.

[0059] Step S207: Determine a block sparse global self-attention weight matrix of the N sampling sub-image features based on the block sparse matrix by the global self-attention network layer.

[0060] Specifically, the second attention sub-network includes a query weight matrix and a key weight matrix. The process by which the computer device determines the block sparse global self-attention weight matrix of the N sampled sub-image features based on the block sparse matrix using the global self-attention network layer may be as follows: the global self-attention network layer constructs a sampled sub-image feature matrix based on the sampled sub-image features included in each of the at least two classification clusters, multiplies the sampled sub-image feature matrix by the query weight matrix to obtain a query matrix, multiplies the sampled sub-image feature matrix by the key weight matrix to obtain a key matrix, determines a block sparse global relevance matrix based on the query matrix, the transpose matrix corresponding to the key matrix, and the block sparse matrix, and performs a normalization process on the block sparse global relevance matrix to obtain the block sparse global self-attention weight matrix.

[0061] Specifically, the above sampling sub-image feature matrix may be expressed as Equation (4).

number

number

number

number

[0062] Specifically, the calculation of the query matrix can be expressed by equation (5).

number

number

number

[0063] Similarly, the calculation of the key matrix can be expressed by equation (6).

number

number

number

[0064] Therefore, the calculation of the block sparse global self-attention weight matrix can be expressed by Equation (7).

number

[0065] For ease of understanding, please also refer to FIG. 6, which is a schematic diagram of the principle of block sparse constraint on global self-attention provided by an embodiment of the present application. As shown in FIG. 6, global self-attention weight matrix 601 is the global self-attention matrix before constraint, which also represents information between sampled sub-image features of different categories. A computer device can obtain block sparse matrix 602 and constrain it to obtain constrained block sparse global self-attention weight matrix 603. As can be seen from FIG. 6, block sparse global self-attention weight matrix 603 only represents information between sampled sub-image features belonging to the same category. Block sparse constraint essentially utilizes category associations to screen global self-attention, focusing on and retaining only attention from the same category and eliminating attention from different categories. Compared to global self-attention weight matrix 601, block sparse global self-attention weight matrix 603 uses instance category information to learn attention and performs appropriate simplified calculations.

[0066] Step S208: The self-attention network layer determines block sparse self-attention corresponding to each sampling sub-image feature based on the at least two classification clusters and the block sparse global self-attention weight matrix.

[0067] Specifically, the second attention sub-network further includes a value weight matrix, and the N sampled sub-image features are represented as sampled sub-image features N awhere a is a positive integer equal to or less than N. The computer device multiplies the sampling sub-image feature matrix and the value weight matrix through a self-attention network layer to obtain a value matrix, and calculates the sampling sub-image feature N a The sampling sub-image feature in the classification cluster to which belongs is defined as the target sampling sub-image feature, and the sampling sub-image feature N a and the target sampling sub-image feature as the target block sparse global self-attention weight; obtain a value vector corresponding to the target sampling sub-image feature from the value matrix as the target value vector; and calculate the sampling sub-image feature N based on the target value vector and the target block sparse global self-attention weight. a where we determine the block sparse self-attention corresponding to the sampling sub-image feature N a is the above sampled sub-image feature matrix

number

[0068] Specifically, the calculation of the value matrix can be expressed by equation (8).

number

number

number

[0069] The calculation of block sparse self-attention can be expressed by Equation (9).

number

number

number

number

number

number

number

number

number

number

number

number

number

number

number

[0070] Step S209: The mean value pooling network layer performs mean value pooling processing on at least two of the block sparse self-attentions to obtain a second feature vector.

[0071] Specifically, the average value pooling process refers to adding at least two block sparse self-attentions and then calculating the average to obtain the resulting vector, i.e., the second feature vector X2.

[0072] Step S210: The first feature vector and the second feature vector are input into a classification sub-network of an image recognition model, and the classification sub-network includes a feature fusion network layer and a classification network layer. The feature fusion network layer performs feature fusion processing on the first feature vector and the second feature vector to obtain a fused feature vector. The classification network layer performs classification processing on the fused feature vector to obtain a classification result for the image to be detected.

[0073] Specifically, the first attention sub-network outputs a first feature vector X1, the second attention sub-network outputs a second feature vector X2, and the two parallel feature vectors undergo feature fusion in the feature fusion network layer. The classification network layer can adopt an MLP classifier, and therefore the final output can be expressed as the following equation (10):

number

[0074] To facilitate understanding of the structure of the image discrimination model in steps S202 to S210, please also refer to FIG. 7, which is a structural schematic diagram of an image discrimination model provided by an embodiment of the present application. As shown in FIG. 7, the image discrimination model 7 includes a first attention sub-network 71, a second attention sub-network 72, and a classification sub-network 73. After obtaining a feature representation subset 700 corresponding to the image to be detected, the computer device sequentially inputs the sub-image features in the feature representation subset 700 into the image discrimination model 7. Here, the feature representation subset 700 includes at least two sub-image features. In the image discrimination model 7, the computer device inputs the feature representation subset 700 into the first attention sub-network 71 and the second attention sub-network 72, respectively. In the first attention subnetwork 71, for each sub-image feature in the feature representation subset 700, a corresponding weight is learned by a parameterized neural network. For example, sub-image feature 711 may be input to parameterized neural network 712, and neural network 712 outputs the weight of sub-image feature 711. For a specific implementation, see step S203 above. Next, in the first attention subnetwork 71, the computer device employs a linear nonlinear attention weighting method to perform effective aggregation on all sub-image features in the feature representation subset 700, ultimately obtaining a first feature vector 713. The aggregation process can see step S204 above. In the second attention subnetwork 72, the computer device first performs unsupervised clustering on the feature representation subset 700 to obtain at least two classification clusters, e.g., classification cluster 721, and determines whether sub-images corresponding to the sub-image features in classification cluster 721 belong to the same category of images. Next, the computer device acquires some sub-image features from each classification cluster as sampling sub-image features, where the clustering and sampling processes can refer to step S206 above.Next, in the second attention sub-network 72, the computer device can obtain a key matrix 723, a query matrix 724, and a value matrix 725 by matrix transforming the sampled sub-image feature matrix 722 composed of the sampled sub-image features. Here, the matrix transformation can refer to the above equations (5), (6), and (8). Specifically, it can be realized by a convolutional network with a 1×1 convolution kernel. Next, in the second attention sub-network 72, the block sparse global self-attention matrix 726 can be determined based on the transpose of the key matrix 723, the query matrix 724, and the block sparse matrix. The determination process can refer to the above step S207. Furthermore, the second feature vector 727 can be determined based on the block sparse global attention matrix 726 and the value matrix 725. The determination process can refer to the above steps S208 and S209. Finally, the computer device inputs the first feature vector 713 and the second feature vector 727 into the classification sub-network 73. In the classification sub-network 73, after feature fusion is performed on the first feature vector 713 and the second feature vector 727, the feature-fused vector is input to a classifier 731, which then outputs a classification result 732, which may include an image normal probability and an image abnormal probability.

[0075] By adopting the method provided in the embodiments of the present application, the first attention sub-network and the second attention sub-network in the image recognition model use two different methods to mine information from the image to be detected, and obtain the first feature vector and the second feature vector. After fusion, the two feature vectors complement and constrain each other, and the fused feature vector can be predicted, and the obtained classification results are highly accurate.

[0076] Further, as shown in FIG. 8, FIG. 8 is a flow diagram of an initial image recognition model training method provided by an embodiment of the present application. Here, the initial image recognition model training method may be executed by a computer device, where the computer device may be the service server 100 shown in FIG. 1a above, or any terminal device in the terminal device group shown in FIG. 1a above, for example, terminal device 10c. Hereinafter, the initial image recognition model training method will be described using an example where the initial image recognition model training method is executed by a computer device. Here, the initial image recognition model training method may include at least the following steps S301 to S305.

[0077] Step S301: Obtain a sample image, perform a feature extraction process on the sample image, and obtain a sample feature representation subset of the sample image, wherein the sample image includes at least two sample sub-images, the sample feature representation subset includes at least two sample sub-image features, and the at least two sample sub-image features correspond one-to-one to the at least two sample sub-images.

[0078] Specifically, for the implementation process of step S301, reference can be made to the description of step S101 in the embodiment corresponding to FIG. 2 above, and detailed description thereof will be omitted here.

[0079] Step S302: Input the at least two sample sub-images into an initial image identification model, generate sample attention weights corresponding to each of the at least two sample sub-image features using the initial image identification model, and perform a weighted aggregation process on the at least two sample sub-image features based on the sample attention weights corresponding to each of the at least two sample sub-image features to obtain a first sample feature vector.

[0080] Specifically, the initial image identification model may include a first initial attention sub-network, and the computer device can use the first initial attention sub-network to generate sample attention weights corresponding to each of at least two sample sub-image features, and perform a weighted aggregation process on the at least two sample sub-image features based on the sample attention weights corresponding to each of the at least two sample sub-image features to obtain a first sample feature vector. For the specific implementation process, please refer to the description of steps S202 to S204 in the embodiment corresponding to Figure 4 above, and detailed explanations will be omitted here.

[0081] Step S303: Using the initial image identification model, perform a clustering sampling process on the at least two sample sub-image features to obtain at least two sample classification clusters, where the sample classification clusters include sample sampling sub-image features; determine sample block sparse self-attentions corresponding to each of the sample sampling sub-image features based on the at least two sample classification clusters and a block sparse matrix; determine a second sample feature vector based on the at least two sample block sparse self-attentions, where the sample block sparse self-attentions corresponding to the sample sampling sub-image features are determined based on the sample sampling sub-image features in the sample classification cluster to which they belong.

[0082] Specifically, the initial image classification model may further include a second initial attention sub-network, which then performs a clustering sampling process on at least two sample sub-image features to obtain sample sampling sub-image features contained in each of the at least two sample classification clusters, determines sample block sparse self-attention corresponding to each sample sampling sub-image feature based on the at least two sample classification clusters and the block sparse matrix, and determines a second sample feature vector based on the at least two sample block sparse self-attention. For the specific implementation process, please refer to the description of steps S205 to S209 in the embodiment corresponding to Figure 4 above, and detailed explanations will be omitted here.

[0083] Step S304: Determine a sample classification result of the sample image according to the first sample feature vector and the second sample feature vector using the initial image identification model.

[0084] Specifically, the initial image identification model may further include an initial classification sub-network, and then the initial classification sub-network determines the sample classification result of the sample image based on the first sample feature vector and the second sample feature vector. For specific implementation, please refer to the description of step S210 in the embodiment corresponding to Figure 4 above, and detailed explanation will be omitted here.

[0085] Step S305: Based on the at least two sample classification clusters, the attention weights corresponding to each of the at least two sample sub-image features, the sample classification results, and the classification labels corresponding to the sample images, perform model parameter adjustment on the initial image identification model to obtain an image identification model used to identify the classification results of the image to be detected.

[0086] Specifically, since the inputs of the first and second attention sub-networks in the final image discrimination model are the same sub-image features, the attention distribution of the first attention sub-network for at least two sub-image features should be consistent with the attention distribution of the second attention sub-network for at least two sub-image features. Therefore, in the process of training the initial image discrimination model, the computer device can first determine a divergence loss value based on the sample attention weights corresponding to at least two sample classification clusters and at least two sample sub-image features, and then determine a classification loss value based on the sample classification results and the classification labels corresponding to the sample images. Finally, the divergence loss value and the classification loss value are weighted together to obtain a total model loss value, and the initial image discrimination model is adjusted based on the total model loss value to obtain the image discrimination model. Here, the divergence loss value is used to ensure that the attention distributions input to similar sub-image features by the two network branches of the final trained image discrimination model are consistent. The classification loss value is used to ensure that the classification results output by the final trained image recognition model are close to the actual results.

[0087] Specifically, the above-mentioned implementation process of determining the divergence loss value based on the sample attention weights corresponding to at least two sample classification clusters and at least two sample sub-image features may be as follows: obtain an i-th sample classification cluster among the at least two sample classification clusters, where i is a positive integer and is less than or equal to the quantity of the at least two sample classification clusters; take the sample sub-image feature included in the i-th sample classification cluster as a target sample sub-image feature; determine a category divergence loss value corresponding to the i-th sample classification cluster based on the sample attention weights corresponding to the target sample sub-image feature and the quantity of the target sample sub-image feature; and add the category divergence loss values ​​corresponding to each sample classification cluster to obtain the divergence loss value.

[0088] When the computer device performs image detection on the sample image, it performs clustering on the sample sub-image features contained in the sample image in the second initial attention sub-network to obtain at least two sample classification clusters, and since the attention levels of the sample sub-image features in the common sample classification clusters in the second initial attention sub-network are the same, the attention levels of the sample sub-image features in the common sample classification clusters in the first initial attention sub-network should also be the same. For example, a sample image includes six sample sub-image features, namely B1, B2, B3, B4, B5, and B6. The sample attention weights generated in the first initial attention sub-network are 0.10, 0.22, 0.11, 0.31, 0.22, and 0.12, respectively. The sample classification clusters generated in the second initial attention sub-network are sample classification cluster 1 {B1, B3, B6} and sample classification cluster 2 {B2, B4, B5}. Therefore, the sample attention weights corresponding to B1, B3, and B4 in sample classification cluster 1 are close to the consensus state and are reasonable. However, the sample attention weight corresponding to B4 in sample classification cluster 2 is significantly higher than B2 and B5, which is not reasonable and therefore needs to be adjusted by the divergence loss value. That is, the attention weights generated by the sample sub-image features in the common sample classification cluster in the first attention sub-network can follow a uniform distribution, so that each sample classification cluster can determine a category divergence loss value. Finally, the category divergence loss values ​​corresponding to each sample classification cluster are added to obtain the divergence loss value.

[0089] Specifically, the implementation process for determining the category divergence loss value corresponding to the i-th sample classification cluster based on the sample attention weights corresponding to the target sample sub-image features and the quantity of the target sample sub-image features may be as follows: obtain a fitted attention distribution composed of the sample attention weights corresponding to the target sample sub-image features, perform a normalization process on the fitted attention weight distribution to obtain a normalized fitted attention distribution, and use the uniform attention distribution corresponding to the quantity of the target sample sub-image features as the attention distribution label; and determine the category divergence loss value corresponding to the i-th sample classification cluster based on the normalized fitted attention distribution and the attention distribution label.

[0090] Suppose the sample attention weights corresponding to the target sample sub-image features are 0.10, 0.12, and 0.11. The constructed fitted attention distribution is [0.10, 0.12, 0.11]. To facilitate the subsequent calculation of the category divergence loss value, the input requires probability. Therefore, the fitted attention distribution is normalized, i.e., the sum of the additions must be 1. The obtained normalized fitted attention distribution is [0.303, 0.363, 0.333]. When the quantity of the target sample sub-image features is 3, the corresponding uniform attention distribution is the attention distribution label, which is [1 / 3, 1 / 3, 1 / 3].

[0091] The process of determining the category divergence loss value corresponding to the i-th sample classification cluster based on the normalized fitted attention distribution and the attention distribution label can be expressed by the following equation (11):

number

number

number

number

[0092] Therefore, the calculation of the divergence loss value can be realized by the following equation (12).

number

number

[0093] Specifically, the total loss value can be determined by the following equation (13).

number

[0094] Specifically, when training the initial image classification model, it can be trained for 100 epochs (periods), the optimizer adopts Adam (optimization algorithm) by default, the initialization learning rate is 1e-4, and the cosine annealing strategy is adopted to adjust the learning rate, and the minimum learning rate is 1e-6.

[0095] By adopting the method provided in the present embodiment, when training the initial image recognition model, additional constraints are placed on the attention distribution of the first initial attention sub-network and the second initial attention sub-network, i.e., the KL divergence loss function is increased in the loss function, ensuring that the attention distributions of the two sub-networks input for similar multiple instances are consistent, and the image detection accuracy of the final trained image recognition model is high.

[0096] 9, which is a structural schematic diagram of an image detection device provided in an embodiment of the present application. The image detection device may be a computer program (including program code) that operates a computer device. For example, the image detection device may be application software that can be used to perform appropriate steps in the image detection method provided in the embodiment of the present application. As shown in FIG. 9, the image detection device 1 may include a feature extraction module 11, a first vector generation module 12, a second vector generation module 13, and a classification module 14.

[0097] The feature extraction module 11 is used to acquire an image of the detection target, perform feature extraction processing on the image of the detection target, and obtain a feature representation subset of the image of the detection target, where the image of the detection target includes at least two sub-images, and the feature representation subset includes at least two sub-image features, and the at least two sub-image features correspond one-to-one to the at least two sub-images; the first vector generation module 12 is used to generate attention weights corresponding to at least two sub-image features, and perform weighted aggregation processing on the at least two sub-image features based on the attention weights to obtain a first feature vector; The second vector generation module 13 performs a clustering sampling process on at least two sub-image features to obtain at least two classification clusters, each of which includes the sampled sub-image features; and determines block sparse self-attention corresponding to each sampled sub-image feature based on the at least two classification clusters and the block sparse matrix, which is used to determine a second feature vector based on the at least two block sparse self-attentions, where the block sparse self-attention corresponding to the sampled sub-image feature is determined based on the sampled sub-image feature in the classification cluster to which it belongs; The classification module 14 is used to determine a classification result of the image to be detected based on the first feature vector and the second feature vector.

[0098] Here, for the specific implementation methods of the feature extraction module 11, the first vector generation module 12, the second vector generation module 13, and the classification module 14, please refer to the relevant description of the embodiment corresponding to Figure 2 above, and detailed explanations will be omitted here.

[0099] Here, the feature extraction module 11 includes a pre-processing unit 111 and a feature extraction unit 112 .

[0100] The pre-processing unit 111 is used to distinguish between background and foreground areas in the image to be detected; The pre-processing unit 111 is further used for performing image segmentation on the image of the detection target according to the background region and the foreground region to obtain a foreground image of the detection target; The pre-processing unit 111 is further used to perform a scaling process on the foreground image to be detected based on the scaling factor to obtain a foreground image to be trimmed; The pre-processing unit 111 is further used to perform a cropping process on the foreground image to be cropped according to the preset length of the sub-image and the preset width of the sub-image to obtain at least two sub-images, where the preset length of the sub-image is less than the length of the foreground image to be cropped, and the preset width of the sub-image is less than the width of the foreground image to be cropped; The feature extraction unit 112 is used to perform image feature extraction processing on at least two sub-images respectively, obtain sub-image features corresponding to each of the at least two sub-images, and determine a feature representation subset of the image to be detected based on the sub-image features corresponding to each of the at least two sub-images.

[0101] Here, for the specific implementation of the pre-processing unit 111 and the feature extraction unit 112, please refer to the relevant description of the embodiment corresponding to FIG. 2 above, and detailed description will be omitted here.

[0102] Here, the first vector generating module 12 includes a first input unit 121 , a weight fitting unit 122 and an aggregation unit 123 .

[0103] The first input unit 121 is used for inputting at least two sub-image features into a first attention sub-network in the image discrimination model, and the first attention sub-network includes a weight learning network layer and a weight summation network layer; The weight fitting unit 122 is used to perform weight fitting processing on the at least two sub-image features through a weight learning network layer, respectively, to obtain attention weights corresponding to the at least two sub-image features; The aggregation unit 123 is used to perform weighting processing on each sub-image feature based on the attention weight using a weighted aggregation network layer to obtain weighted sub-image features corresponding to each sub-image feature, and to perform aggregation processing on the weighted sub-image features corresponding to each of the at least two sub-image features to obtain a first feature vector.

[0104] Here, for the specific implementation manner of the first input unit 121, the weight fitting unit 122, and the aggregation unit 123, please refer to the relevant description of the embodiment corresponding to Figure 4 above, and detailed description will be omitted here.

[0105] Here, the second vector generation module 13 includes a second input unit 131, a clustering sampling unit 132, a global self-attention determination unit 133, a self-attention determination unit 134, and an average value pooling unit 135.

[0106] The second input unit 131 is used for inputting at least two sub-image features into a second attention sub-network in the image discrimination model, and the second attention sub-network includes a clustering sampling network layer, a global self-attention network layer, a self-attention network layer, and an average pooling network layer; The clustering sampling unit 132 is used to perform clustering sampling on the at least two sub-image features through a clustering sampling network layer to obtain at least two classification clusters including the sampled sub-image features, and the sum of the quantity of the sampled sub-image features included in each of the at least two classification clusters is N, where N is a positive integer less than the quantity of the at least two sub-image features; The global self-attention determination unit 133 is used to determine a block sparse global self-attention weight matrix of the N sampling sub-image features according to the block sparse matrix through a global self-attention network layer; The self-attention determination unit 134 is used to determine block sparse self-attention corresponding to each sampling sub-image feature according to at least two classification clusters and the block sparse global self-attention weight matrix through a self-attention network layer; The mean value pooling unit 135 is used to perform mean value pooling processing on at least two block sparse self-attentions through a mean value pooling network layer to obtain a second feature vector.

[0107] Here, for the specific implementation methods of the second input unit 131, the clustering sampling unit 132, the global self-attention determination unit 133, the self-attention determination unit 134, and the average value pooling unit 135, please refer to the relevant description of the embodiment corresponding to Figure 4 above, and detailed explanations will be omitted here.

[0108] Here, the clustering sampling unit 132 includes a clustering subunit 1321 and a sampling subunit 1322 .

[0109] The clustering subunit 1321 is used to perform clustering processing on at least two sub-image features through a clustering sampling network layer to obtain at least two classification clusters; The sampling subunit 1322 is used to obtain a k-th classification cluster among the at least two classification clusters, where k is a positive integer, and the k-th classification cluster includes at least one clustering sub-image feature; The sampling subunit 1322 is further used for obtaining a vector distance between each of the at least one clustering sub-image features and the cluster center of the k-th classification cluster as a reference distance; The sampling subunit 1322 is further used to sequentially obtain h clustering sub-image features in at least one clustering sub-image feature based on the reference distance, and set the h clustering sub-image features as sampling sub-image features included in the kth classification cluster, where h is a positive integer and h is less than or equal to the quantity of the at least one clustering sub-image feature.

[0110] Here, for the specific implementation of the clustering subunit 1321 and the sampling subunit 1322, please refer to the relevant description of the embodiment corresponding to FIG. 4 above, and detailed description will be omitted here.

[0111] Here, the second attention sub-network includes a query weight matrix and a key weight matrix.

[0112] The global self-attention determination unit 133 includes a first initialization subunit 1331 , a matrix determination subunit 1332 , and a normalization subunit 1333 .

[0113] The first initialization subunit 1331 is used to construct a sampling sub-image feature matrix by a global self-attention network layer according to the sampling sub-image features included in each of the at least two classification clusters; The first initialization sub-unit 1331 is further used for multiplying the sampling sub-image feature matrix and the query weight matrix to obtain a query matrix, and multiplying the sampling sub-image feature matrix and the key weight matrix to obtain a key matrix; The matrix determination subunit 1332 is used to determine a block sparse global relevance matrix according to the query matrix, the transposed matrix corresponding to the key matrix, and the block sparse matrix; The normalization sub-unit 1333 is used to perform normalization processing on the block sparse global relevance matrix to obtain the block sparse global self-attention weight matrix.

[0114] Here, for the specific implementation methods of the first initialization subunit 1331, the matrix determination subunit 1332, and the normalization subunit 1333, please refer to the relevant description of the embodiment corresponding to Figure 4 above, and detailed explanations will be omitted here.

[0115] Here, the second attention sub-network further includes a value weight matrix, and the N sampled sub-image features are represented as sampled sub-image feature Nj where j is a positive integer less than or equal to N, The self-attention determination unit 134 includes a target acquisition subunit 1341 and a determination subunit 1342 .

[0116] The target acquisition subunit 1341 is used to multiply the sampling sub-image feature matrix and the value weight matrix through the self-attention network layer to obtain a value matrix; The target acquisition subunit 1341 further comprises: j The sampling sub-image feature in the classification cluster to which the target sub-image feature belongs is used as the target sampling sub-image feature; The target acquisition subunit 1341 further selects the sampled sub-image features N from the block sparse global self-attention weight matrix. j and the target sampling sub-image feature, and obtain the block sparse global self-attention weight as the target block sparse global self-attention weight; The target obtaining subunit 1341 is further used for obtaining a value vector corresponding to the target sampling sub-image feature from the value matrix as a target value vector; The determining subunit 1342 determines the sampling sub-image feature N based on the target value vector and the target block sparse global self-attention weights. j is used to determine the block sparse self-attention corresponding to

[0117] Here, for the specific implementation manner of the target acquisition subunit 1341 and the determination subunit 1342, please refer to the relevant description of the embodiment corresponding to FIG. 4 above, and detailed description will be omitted here.

[0118] Here, the classification module 14 includes a third input unit 141 , a feature fusion unit 142 , and a classification unit 143 .

[0119] The third input unit 141 is used for inputting the first feature vector and the second feature vector into a classification sub-network of the image recognition model, and the classification sub-network includes a feature fusion network layer and a classification network layer; The feature fusion unit 142 is used to perform feature fusion processing on the first feature vector and the second feature vector through a feature fusion network layer to obtain a fused feature vector; The classification unit 143 is used to perform classification processing on the fused feature vector through a classification network layer to obtain a classification result of the image to be detected.

[0120] Here, for the specific implementation methods of the third input unit 141, the feature fusion unit 142, and the classification unit 143, please refer to the relevant description of the embodiment corresponding to Figure 4 above, and detailed explanations will be omitted here.

[0121] Referring to FIG. 10, FIG. 10 is a structural schematic diagram of a computer device provided by an embodiment of the present application. As shown in FIG. 10, the image detection device 1 in the embodiment corresponding to FIG. 10 can be applied to a computer device 1000, which may include a processor 1001, a network interface 1004, and a memory 1005. The computer device 1000 may also include a user interface 1003 and at least one communication bus 1002. The communication bus 1002 is used to realize communication between these assemblies. The user interface 1003 may include a display screen and a keyboard. Optionally, the user interface 1003 may further include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (e.g., a Wi-Fi interface). The memory 1005 may be a high-speed RAM memory or a non-volatile memory, such as at least one magnetic disk memory. The memory 1005 may optionally be at least one storage device remote from the processor 1001. As shown in Figure 10, the memory 1005, which serves as a computer-readable storage medium, may include an operating system, a network communication module, a user interface module, and a device control application program.

[0122] In the computer device 1000 shown in FIG. 10, the network interface 1004 can provide a network communication network element, the user interface 1003 is mainly used to provide an input interface for the user, and the processor 1001 is used to call the device control application program stored in the memory 1005, thereby realizing the image detection method provided by the embodiment of the present application.

[0123] It should be understood that the computer device 1000 described in the present embodiment can execute the image detection method described in the embodiment corresponding to either of Figures 2 and 4 above, and a detailed description thereof will be omitted here, as will a detailed description of the beneficial effects of employing the same method.

[0124] It should be noted that the present embodiment also provides a computer-readable storage medium, storing a computer program executed by the image detection device 1 mentioned above, the computer program including program instructions. When the processor executes the program instructions, it can execute the image detection method described in the embodiment corresponding to either FIG. 2 or FIG. 4 above. Therefore, a detailed description thereof will be omitted here. Also, a detailed description of the beneficial effects of the same method will be omitted. For technical details not disclosed in the computer-readable storage medium embodiment related to the present application, please refer to the description of the method embodiment of the present application.

[0125] Further, please refer to Figure 11, which is a structural schematic diagram of another image detection device provided by an embodiment of the present application. The image detection device 2 may be a computer program (including program code) running in a computer device. For example, the image detection device 2 may be an application software, and the device may be used to perform the processing steps of the method provided by the embodiment of the present application. As shown in Figure 11, the image detection device 2 may include a sample feature extraction module 21, a first sample vector generation module 22, a second sample vector generation module 23, a sample classification module 24, and a training module 25.

[0126] The sample feature extraction module 21 is used to obtain a sample image, perform feature extraction processing on the sample image, and obtain a sample feature representation subset of the sample image, where the sample image includes at least two sample sub-images, and the sample feature representation subset includes at least two sample sub-image features, and the at least two sample sub-image features correspond one-to-one to the at least two sample sub-images; The first sample vector generation module 22 is used to input at least two sample sub-images into an initial image discrimination model, generate sample attention weights corresponding to the at least two sample sub-image features through the initial image discrimination model, and perform a weighted aggregation process on the at least two sample sub-image features based on the sample attention weights corresponding to the at least two sample sub-image features to obtain a first sample feature vector; The second sample vector generation module 23 performs a clustering sampling process on at least two sample sub-image features according to the initial image discrimination model to obtain at least two sample classification clusters, each of which includes the sample sampling sub-image features; and determines sample block sparse self-attentions corresponding to each sample sampling sub-image feature according to the at least two sample classification clusters and the block sparse matrix, which are used to determine a second sample feature vector according to the at least two sample block sparse self-attentions, and the sample block sparse self-attentions corresponding to the sample sampling sub-image features are determined according to the sample sampling sub-image features in the sample classification cluster to which they belong; The sample classification module 24 is used to determine a sample classification result of the sample image according to the first sample feature vector and the second sample feature vector according to the initial image identification model; The training module 25 is used to adjust model parameters for the initial image identification model based on at least two sample classification clusters, attention weights corresponding to each of at least two sample sub-image features, the sample classification results, and the classification labels corresponding to the sample images, to obtain an image identification model used to identify the classification results of the image to be detected.

[0127] Here, for the specific implementation methods of the sample feature extraction module 21, the first sample vector generation module 22, the second sample vector generation module 23, the sample classification module 24, and the training module 25, please refer to the relevant description of the embodiment corresponding to Figure 8 above, and detailed explanations will be omitted here.

[0128] Here, the training module 25 includes a divergence loss value determining unit 251, a classification loss value determining unit 252, a weighted sum obtaining unit 253, and a model adjusting unit 254.

[0129] The divergence loss value determining unit 251 is used to determine a divergence loss value according to the at least two sample classification clusters and the sample attention weights corresponding to each of the at least two sample sub-image features; The classification loss value determination unit 252 is used to determine a classification loss value according to the sample classification result and the classification label corresponding to the sample image; The weighted sum obtaining unit 253 is used to obtain a weighted sum of the divergence loss value and the classification loss value to obtain a model total loss value; The model adjustment unit 254 is used to perform model parameter adjustment on the initial image identification model based on the model total loss value to obtain an image identification model. Here, for the specific implementation of the divergence loss value determination unit 251, the classification loss value determination unit 252, the weighted sum acquisition unit 253, and the model adjustment unit 254, please refer to the relevant description of the embodiment corresponding to Figure 8 above, and detailed description will be omitted here.

[0130] Here, the divergence loss value determining unit 251 includes an acquisition subunit 2511 , a category loss value determining subunit 2512 , and a total loss value determining subunit 2513 .

[0131] The acquiring subunit 2511 is used to acquire an i-th sample classification cluster in the at least two sample classification clusters, where i is a positive integer, and i is less than or equal to the number of the at least two sample classification clusters; The obtaining subunit 2511 is further used for determining the sample sub-image features included in the i-th sample classification cluster as the target sample sub-image features; The category loss value determining subunit 2512 is used to determine a category divergence loss value corresponding to the i-th sample classification cluster according to the sample attention weights corresponding to the target sample sub-image features and the quantity of the target sample sub-image features; The total loss value determination subunit 2513 sums the category divergence loss values ​​corresponding to each sample classification cluster to obtain a divergence loss value.

[0132] Here, the category loss value determination subunit 2512 is specifically used for obtaining a fitting attention distribution composed of sample attention weights corresponding to target sample sub-image features, performing a normalization process on the fitting attention weight distribution to obtain a normalized fitting attention distribution, taking the uniform attention distribution corresponding to the quantity of the target sample sub-image features as the attention distribution label, and determining a category divergence loss value corresponding to the i-th sample classification cluster based on the normalized fitting attention distribution and the attention distribution label.

[0133] Here, for the specific implementation methods of the acquisition subunit 2511, the category loss value determination subunit 2512, and the total loss value determination subunit 2513, please refer to the relevant description of the embodiment corresponding to Figure 8 above, and detailed description will be omitted here.

[0134] Further, please refer to FIG. 12, which is a structural schematic diagram of another computer device provided by an embodiment of the present application. As shown in FIG. 12, the image detection device 2 in the embodiment corresponding to FIG. 11 can be applied to a computer device 2000, which may include a processor 2001, a network interface 2004, and a memory 2005. The computer device 2000 further includes a user interface 2003 and at least one communication bus 2002. The communication bus 2002 is used to realize connection and communication between these assemblies. The user interface 2003 may include a display screen and a keyboard, and optionally, the user interface 2003 may further include a standard wired interface or a wireless interface. The network interface 2004 may optionally include a standard wired interface or a wireless interface (e.g., a Wi-Fi interface). The memory 2005 may be a high-speed RAM memory or a non-volatile memory, such as at least one magnetic disk memory. The memory 2005 may optionally be at least one storage device remote from the processor 2001. As shown in Figure 12, the memory 2005, which serves as a computer-readable storage medium, may include an operating system, a network communication module, a user interface module, and a device control application program.

[0135] In the computer device 2000 shown in FIG. 12, the network interface 2004 can provide network communication functions, the user interface 2003 is mainly used to provide an input interface for the user, and the processor 2001 can be used to call the device control application program stored in the memory 2005, thereby realizing the method for training the initial image recognition model provided in the embodiments of the present application.

[0136] It should be understood that the computer device 2000 described in the present embodiment can execute the description of the access control method in each of the preceding embodiments, and can also execute the description of the image detection device 2 in the embodiment corresponding to Figure 11 in the preceding embodiment, and detailed descriptions thereof will be omitted here. Also, detailed descriptions of the beneficial effects of employing the same method will be omitted.

[0137] It should be noted that the present embodiment also provides a computer-readable storage medium, storing a computer program executed by the image detection device 2 mentioned above. When the processor loads and executes the computer program, it can execute the access control method described in any of the previous embodiments. Therefore, a detailed description thereof will be omitted here. Also, a detailed description of the beneficial effects of the same method will be omitted. For technical details not disclosed in the computer-readable storage medium embodiment related to the present application, please refer to the description of the method embodiment of the present application.

[0138] The computer-readable storage medium may be the image detection device provided in any of the above embodiments or the internal storage unit of the computer device, such as the hard disk or memory of the computer device. The computer-readable storage medium may also be an external storage device of the computer device, such as a removable hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc., located on the computer device. Furthermore, the computer-readable storage medium may not only include the internal storage unit of the computer device, but also an external storage device. The computer-readable storage medium is used to store the computer program and other programs and data required by the computer device. The computer-readable storage medium may also be used to temporarily store data that has already been output or is to be output.

[0139] It should also be noted that the present embodiment further provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium, and a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions to cause the computer device to perform the method provided by any of the embodiments corresponding to the preceding sentence.

[0140] The terms "first," "second," etc. in the description, claims, and drawings of the present application are used to distinguish between different objects, not to describe a particular order. Furthermore, the term "comprises" and any variations thereof are intended to cover a non-exclusive inclusion. For example, a process, method, apparatus, product, or device that includes a series of steps or units is not limited to the steps or modules already listed, but may optionally further include unlisted steps or modules, or may optionally further include other step units inherent to the process, method, apparatus, product, or device.

[0141] As those skilled in the art can appreciate, the units and algorithm steps of each example described in combination with the embodiments disclosed herein may be realized in electronic hardware, computer software, or a combination of both. In order to clearly explain the compatibility between hardware and software, the above description has generally described the configurations and steps of each example according to network elements. Whether these network elements are implemented in a hardware or software manner is determined by the specific application and design constraints of the technical solution. Those skilled in the art may implement the described network elements using different methods for each specific application, but such implementation should not be considered to go beyond the scope of the present application.

[0142] The above are merely preferred embodiments of the present application, and of course, they cannot limit the scope of the present application. Therefore, equivalent changes made according to the claims of the present application still fall within the scope covered by the present application. [Explanation of symbols]

[0143] 10 Terminal Equipment 11 Feature Extraction Module 12 First Vector Generation Module 13 Second Vector Generation Module 14 Classification Module 21 Sample Feature Extraction Module 22 First sample vector generation module 23 Second sample vector generation module 24 Sample Classification Modules 25 Training Modules 50 Feature Extractor RestNet 71 First Attention Subnetwork 72 Second Attention Sub-Network 73 Classification Subnetworks 100 Service Servers 111 Pretreatment Unit 112 Feature Extraction Unit 121 First input unit 122 Fitting Unit 123 tally units 131 Second input unit 132 clustering sampling units 133 Global Self-Attention Decision Unit 134 Self-Attention Decision Unit 135 Mean Pooling Units 141 Third Input Unit 142 Feature Fusion Unit 143 classification units 200 Terminal Equipment 251 Divergence Loss Value Determination Unit 252 Classification Loss Value Determination Unit 253 Weighted Sum Acquisition Units 254 Model Adjustment Unit 300 Patient Management Applications 301 Colorectal Pathology Images 400 Service Server 401 Feature Representation Subset 402 Image Recognition Model 402 Training Image Classification Model 403 First feature vector 404 Second feature vector 405 Classification result 601 Queue 602 Block Sparse Matrices 603 Queue 700 feature representation subset 712 Neural Networks 713 First feature vector 721 Classification Clusters 722 Sampling sub-image feature matrix 723 Key Matrix 724 Queue 725-value matrix 726 Block Sparse Global Self-Attention Matrix 726 Block Sparse Global Attention Matrix 727 Second feature vector 731 Classifier 732 Classification results 1000 Computer Equipment 1001 processor 1002 communication bus 1003 User Interface 1004 Network Interface 1005 memory 1321 Clustering Subunit 1322 Sampling Subunit 1331 First Initialization Subunit 1332 Matrix Determination Subunit 1333 normalization subunit 1341 Target Acquisition Subunit 1342 Decision Subunit 2000 Computer Equipment 2001 processor 2002 communication bus 2003 User Interface 2004 Network Interface 2005 Memory 2511 Acquisition Subunit 2512 Category Loss Value Determination Subunit 2513 Total Loss Value Determination Subunit 3000 images 3001 Foreground area 3002 Background area 3003 Foreground image 3004 sub-images 3005 sub-images 3006 sub-images 3007 Feature Extractor 3008 Feature Representation Subset 4021 First Attention Subnetwork 4022 Second Attention Sub-Network 4023 Classification Subnetwork

Claims

1. An image detection method in which a computer device performs each step, comprising: acquiring an image of the target to be detected, and performing a feature extraction process on the image of the target to obtain a feature representation subset of the image of the target to be detected, wherein the image of the target to be detected includes at least two sub-images, and the feature representation subset includes at least two sub-image features, and the at least two sub-image features correspond one-to-one to the at least two sub-images; generating an attention weight corresponding to each of the at least two sub-image features, and performing a weighted aggregation process on the at least two sub-image features based on the attention weight to obtain a first feature vector; performing a clustering sampling process on the at least two sub-image features to obtain at least two classification clusters, each of which includes the sampled sub-image features; determining block sparse self-attentions corresponding to each of the sampled sub-image features based on the at least two classification clusters and a block sparse matrix; and determining a second feature vector based on the at least two block sparse self-attentions, wherein the block sparse self-attentions corresponding to each of the sampled sub-image features are determined based on the sampled sub-image features in the classification cluster to which the block sparse self-attentions belong; determining a classification result of the image to be detected based on the first feature vector and the second feature vector.

2. The step of performing a feature extraction process on the image of the detection target and obtaining a feature representation subset of the image of the detection target includes: identifying a background region and a foreground region in the image of the detection target; performing image segmentation on the image of the detection target based on the background region and the foreground region to obtain a foreground image of the detection target; performing a scaling process on the foreground image to be detected based on the scaling factor to obtain a foreground image to be trimmed; performing a cropping process on the foreground image to be cropped based on a preset length of a sub-image and a preset width of a sub-image to obtain the at least two sub-images, wherein the preset length of the sub-image is less than the length of the foreground image to be cropped, and the preset width of the sub-image is less than the width of the foreground image to be cropped; 2. The method of claim 1, further comprising: performing an image feature extraction process on each of the at least two sub-images to obtain sub-image features corresponding to each of the at least two sub-images; and determining a feature representation subset of the image to be detected based on the sub-image features corresponding to each of the at least two sub-images.

3. generating attention weights corresponding to the at least two sub-image features, and performing a weighted aggregation process on the at least two sub-image features based on the attention weights to obtain a first feature vector; inputting the at least two sub-image features into a first attention sub-network in an image discrimination model, the first attention sub-network including a weight learning network layer and a weight summation network layer; performing a weight fitting process on the at least two sub-image features by the weight learning network layer to obtain attention weights corresponding to the at least two sub-image features; 2. The method of claim 1, further comprising: using the weighted aggregation network layer to perform a weighting process on each of the sub-image features based on the attention weights to obtain weighted sub-image features corresponding to each of the sub-image features; and performing an aggregation process on the weighted sub-image features corresponding to each of the at least two sub-image features to obtain the first feature vector.

4. performing a clustering sampling process on the at least two sub-image features to obtain at least two classification clusters, each of the classification clusters including the sampled sub-image features; determining block sparse self-attentions corresponding to each of the sampled sub-image features based on the at least two classification clusters and a block sparse matrix; and determining a second feature vector based on the at least two block sparse self-attentions; inputting the at least two sub-image features into a second attention sub-network in an image discrimination model, the second attention sub-network including a clustering sampling network layer, a global self-attention network layer, a self-attention network layer, and an average pooling network layer; performing a clustering sampling process on the at least two sub-image features by the clustering sampling network layer to obtain the at least two classification clusters, wherein the classification clusters include the sampled sub-image features, and the sum of the number of sampled sub-image features included in each of the at least two classification clusters is N, where N is a positive integer less than the number of the at least two sub-image features; determining a block-sparse global self-attention weight matrix for the N sampled sub-image features based on the block-sparse matrix by the global self-attention network layer; determining, by the self-attention network layer, block sparse self-attention corresponding to each of the sampled sub-image features based on the at least two classification clusters and the block sparse global self-attention weight matrix; and performing an average pooling process on at least two of the block sparse self-attention features by the average pooling network layer to obtain a second feature vector.

5. The step of performing a clustering sampling process on the at least two sub-image features by the clustering sampling network layer to obtain the at least two classification clusters includes: performing a clustering process on the at least two sub-image features by the clustering sampling network layer to obtain the at least two classification clusters; obtaining a k-th classification cluster among the at least two classification clusters, where k is a positive integer, and the k-th classification cluster includes at least one clustering sub-image feature; obtaining a vector distance between each of the at least one clustering sub-image feature and the cluster center of the k-th classification cluster as a reference distance; 5. The method of claim 4, further comprising: sequentially obtaining h clustering sub-image features in the at least one clustering sub-image feature based on the reference distance, and setting the h clustering sub-image features as sampling sub-image features included in the kth classification cluster, wherein h is a positive integer and h is less than or equal to a quantity of the at least one clustering sub-image feature.

6. The second attention sub-network includes a query weight matrix and a key weight matrix, and the step of determining a block-sparse global self-attention weight matrix of the N sampled sub-image features based on the block-sparse matrix by the global self-attention network layer includes: constructing, by the global self-attention network layer, a sampled sub-image feature matrix based on the sampled sub-image features contained in each of the at least two classification clusters; multiplying the sampled sub-image feature matrix with the query weight matrix to obtain a query matrix, and multiplying the sampled sub-image feature matrix with the key weight matrix to obtain a key matrix; determining a block sparse global relevance matrix based on the query matrix, a transpose matrix corresponding to the key matrix, and the block sparse matrix; and performing a normalization process on the block sparse global relevance matrix to obtain the block sparse global self-attention weight matrix.

7. The second attention sub-network further includes a value weight matrix, and the N sampled sub-image features are sampled sub-image features N ｊ wherein j is a positive integer equal to or less than N, and the step of determining, by the self-attention network layer, block sparse self-attention corresponding to each of the sampled sub-image features based on the at least two classification clusters and the block sparse global self-attention weight matrix includes: multiplying the sampled sub-image feature matrix and the value weight matrix by the self-attention network layer to obtain a value matrix; The sampled sub-image feature N ｊ a step of setting a sampling sub-image feature in a classification cluster to which the target sampling sub-image feature belongs; From the block sparse global self-attention weight matrix, the sampled sub-image features N ｊ and the target sampling sub-image feature as a target block sparse global self-attention weight; obtaining a value vector corresponding to the target sampling sub-image feature from the value matrix as a target value vector; Based on the target value vector and the target block sparse global self-attention weights, the sampled sub-image feature N ｊ and determining block sparse self-attention corresponding to

8. The step of determining a classification result of the image of the detection target based on the first feature vector and the second feature vector includes: inputting the first feature vector and the second feature vector into a classification sub-network of an image discrimination model, the classification sub-network including a feature fusion network layer and a classification network layer; performing a feature fusion process on the first feature vector and the second feature vector by the feature fusion network layer to obtain a fused feature vector; The method of claim 1 , further comprising: performing a classification process on the fused feature vector using the classification network layer to obtain a classification result of the image to be detected.

9. 1. An image detection method implemented by a computing device, comprising: obtaining a sample image and performing a feature extraction process on the sample image to obtain a sample feature representation subset of the sample image, wherein the sample image includes at least two sample sub-images, and the sample feature representation subset includes at least two sample sub-image features, and the at least two sample sub-image features have a one-to-one correspondence with the at least two sample sub-images; inputting the at least two sample sub-images into an initial image discrimination model, generating sample attention weights corresponding to each of the at least two sample sub-image features using the initial image discrimination model, and performing a weighted aggregation process on the at least two sample sub-image features based on the sample attention weights corresponding to each of the at least two sample sub-image features to obtain a first sample feature vector; performing a clustering sampling process on the at least two sample sub-image features using the initial image discrimination model to obtain at least two sample classification clusters, each of which includes sample sampling sub-image features; determining sample block sparse self-attentions corresponding to each of the sample sampling sub-image features based on the at least two sample classification clusters and a block sparse matrix; and determining a second sample feature vector based on the at least two sample block sparse self-attentions, wherein the sample block sparse self-attentions corresponding to each of the sample sampling sub-image features are determined based on the sample sampling sub-image features in the sample classification cluster to which the sample sampling sub-image features belong; determining a sample classification result of the sample image based on the first sample feature vector and the second sample feature vector using the initial image discrimination model; An image detection method comprising: a step of adjusting model parameters of the initial image identification model based on the at least two sample classification clusters, attention weights corresponding to each of the at least two sample sub-image features, the sample classification results, and classification labels corresponding to the sample images, to obtain an image identification model used to identify the classification results of the image to be detected.

10. The step of adjusting model parameters of the initial image discrimination model based on the at least two sample classification clusters, attention weights corresponding to each of the at least two sample sub-image features, the sample classification results, and classification labels corresponding to the sample images to obtain an image discrimination model used to discriminate the classification results of the image to be detected includes: determining a divergence loss value based on the at least two sample classification clusters and sample attention weights corresponding to each of the at least two sample sub-image features; determining a classification loss value based on the sample classification results and classification labels corresponding to the sample images; performing a weighted sum of the divergence loss value and the classification loss value to obtain a total model loss value; and performing model parameter adjustment on the initial image discrimination model based on the total model loss value to obtain the image discrimination model.

11. determining a divergence loss value based on the at least two sample classification clusters and sample attention weights corresponding to each of the at least two sample sub-image features; obtaining an i-th sample classification cluster in the at least two sample classification clusters, where i is a positive integer and is less than or equal to a quantity of the at least two sample classification clusters; determining a sample sub-image feature included in the i-th sample classification cluster as a target sample sub-image feature; determining a category divergence loss value corresponding to the i-th sample classification cluster based on sample attention weights corresponding to the target sample sub-image features and the quantity of the target sample sub-image features; and adding the category divergence loss values ​​corresponding to each sample classification cluster to obtain the divergence loss value.

12. determining a category divergence loss value corresponding to the i-th sample classification cluster based on sample attention weights corresponding to the target sample sub-image features and the quantity of the target sample sub-image features, obtaining a fitted attention distribution composed of sample attention weights corresponding to the target sample sub-image features; performing a normalization process on the fitted attention distribution to obtain a normalized fitted attention distribution; A uniform attention distribution corresponding to the quantity of the target sample sub-image features is taken as an attention distribution label; and determining a category divergence loss value corresponding to the i-th sample classification cluster based on the normalized fitted attention distribution and the attention distribution label.

13. 1. An image detection device, comprising: a feature extraction module used to acquire an image of a target object, perform a feature extraction process on the image of the target object, and obtain a feature representation subset of the image of the target object, wherein the image of the target object includes at least two sub-images, and the feature representation subset includes at least two sub-image features, and the at least two sub-image features correspond one-to-one to the at least two sub-images; a first vector generation module used to generate attention weights corresponding to the at least two sub-image features, and perform a weighted aggregation process on the at least two sub-image features based on the attention weights to obtain a first feature vector; a second vector generation module for performing a clustering sampling process on the at least two sub-image features to obtain at least two classification clusters, each of which includes the sampled sub-image features; determining block sparse self-attentions corresponding to each of the sampled sub-image features based on the at least two classification clusters and a block sparse matrix; and determining a second feature vector based on the at least two block sparse self-attentions, wherein the block sparse self-attentions corresponding to each of the sampled sub-image features are determined based on the sampled sub-image features in the classification cluster to which the block sparse self-attentions belong; a classification module used to determine a classification result of the target image based on the first feature vector and the second feature vector.

14. 1. An image detection device, comprising: a sample feature extraction module used to acquire a sample image, perform a feature extraction process on the sample image, and obtain a sample feature representation subset of the sample image, wherein the sample image includes at least two sample sub-images, and the sample feature representation subset includes at least two sample sub-image features, and the at least two sample sub-image features correspond one-to-one with the at least two sample sub-images; a first sample vector generation module, which is used to input the at least two sample sub-images into an initial image discrimination model, generate sample attention weights corresponding to each of the at least two sample sub-image features using the initial image discrimination model, and perform a weighted aggregation process on the at least two sample sub-image features based on the sample attention weights corresponding to each of the at least two sample sub-image features to obtain a first sample feature vector; a second sample vector generation module for performing a clustering sampling process on the at least two sample sub-image features using the initial image discrimination model to obtain at least two sample classification clusters, each of which includes sample sampling sub-image features; determining sample block sparse self-attentions corresponding to each of the sample sampling sub-image features based on the at least two sample classification clusters and a block sparse matrix; and determining a second sample feature vector based on the at least two sample block sparse self-attentions, wherein the sample block sparse self-attentions corresponding to each of the sample sampling sub-image features are determined based on the sample sampling sub-image features in the sample classification cluster to which the sample sampling sub-image features belong; a sample classification module used to determine a sample classification result of the sample image based on the first sample feature vector and the second sample feature vector according to the initial image discrimination model; an image detection device comprising: a training module used to adjust model parameters of the initial image identification model based on the at least two sample classification clusters, attention weights corresponding to each of the at least two sample sub-image features, the sample classification results, and classification labels corresponding to the sample images, thereby obtaining an image identification model used to identify the classification results of the image to be detected.

15. A computer device including a processor, a memory, and a network interface, The processor is connected to the memory and the network interface, wherein the network interface is used to provide data communication functions, the memory is used to store program code, and the processor calls the program code, thereby performing the method of any one of claims 1 to 12.

16. A computer program comprising instructions which, when executed by a processor, cause the processor to carry out a method according to any one of claims 1 to 12.

Images