Image classification methods, image classification devices, electronic devices and storage media

Abstract

This application discloses an image classification method, image classification device, electronic device, and storage medium, belonging to the field of computer processing technology. The method includes: segmenting a target image to be classified into multiple target image blocks; merging the multiple target image blocks into at least two target image regions based on the correlation between them, each target image region including at least one target image block; determining the image features of each target image region; and determining the category of the target image based on the image features of each target image region. This method achieves the goal of determining the category of the target image by comprehensively considering the correlation between target image blocks, improving the accuracy of image classification results. Furthermore, merging multiple target image blocks into at least two target image regions reduces the number of blocks and improves the efficiency of image classification processing.

Description

Technical Field

[0001] This application relates to the field of computer processing technology, and in particular to an image classification method, an image classification device, an electronic device, and a storage medium. Background Technology

[0002] In the field of computer processing technology, images are a commonly used information carrier, and image classification techniques can be used to classify and process images. Image classification technology is a technique that determines the category of an image or the category of objects within an image based on the features reflected in the image information.

[0003] In related technologies, there are some ultra-large images, such as remote sensing images and whole-slide images (WSI), whose dimensions are typically greater than or equal to 10,000 pixels by 10,000 pixels. Since computer computing resources are limited, for ultra-large images, it is necessary to segment the image into multiple image blocks of the same size and determine the image features of each block. For each image block, the image block is analyzed and processed based on its image features to obtain the analysis and processing results. Then, based on the analysis and processing results of each image block, the image category is determined.

[0004] The aforementioned techniques analyze and process each image patch independently, which affects the accuracy of image classification results. Furthermore, the large number of image patches reduces the efficiency of image classification processing. Summary of the Invention

[0005] This application provides an image classification method, an image classification device, an electronic device, and a storage medium, which can be used to solve the problems of low accuracy of image classification results and low efficiency of image classification processing in related technologies. The technical solution includes the following contents.

[0006] On one hand, embodiments of this application provide an image classification method, the method comprising:

[0007] The target image to be classified is divided into multiple target image patches;

[0008] Based on the correlation between the multiple target image blocks, the multiple target image blocks are merged into at least two target image regions, and the target image region includes at least one target image block;

[0009] Determine the image features of each target image region;

[0010] The category of the target image is determined based on the image features of each target image region.

[0011] On the other hand, embodiments of this application provide an image classification apparatus, the apparatus comprising:

[0012] The segmentation module is used to segment the target image to be classified into multiple target image patches;

[0013] The merging module is used to merge the multiple target image blocks into at least two target image regions based on the correlation between the multiple target image blocks, wherein the target image regions include at least one target image block;

[0014] The determination module is used to determine the image features of each target image region;

[0015] The determining module is further configured to determine the category of the target image based on the image features of each target image region.

[0016] In one possible implementation, the merging module is used to determine the position of each target image block in the target image; based on the position and quantity threshold of each target image block in the target image, the plurality of target image blocks are merged into at least two target image regions, wherein the number of target image blocks included in the target image region is less than or equal to the quantity threshold, and any two target image blocks included in the target image region are adjacent in the target image.

[0017] In one possible implementation, the merging module is used to determine the category of each target image block; for any two target image blocks, if the categories of the two target image blocks are the same, then the two target image blocks are merged into one target image region; if the categories of the two target image blocks are different, then the two target image blocks are merged into two target image regions.

[0018] In one possible implementation, the merging module is used to acquire the image features of each target image block, wherein the image features of any target image block are used to describe the texture information of the target image block; and to determine the category of each target image block based on the image features of each target image block.

[0019] In one possible implementation, the device further includes:

[0020] The segmentation module is also used to segment the sample image into multiple sample image blocks;

[0021] The acquisition module is used to acquire the target feature extraction network based on the multiple sample image blocks;

[0022] The merging module is used to obtain the image features of each target image block based on the target feature extraction network.

[0023] In one possible implementation, the acquisition module is configured to perform a first data augmentation process on each sample image block to obtain each sample image block after the first data augmentation process; perform a second data augmentation process on each sample image block to obtain each sample image block after the second data augmentation process; acquire image features of each sample image block after the first data augmentation process and image features of each sample image block after the second data augmentation process based on an initial feature extraction network; and adjust the initial feature extraction network based on the image features of each sample image block after the first data augmentation process and image features of each sample image block after the second data augmentation process to obtain the target feature extraction network.

[0024] In one possible implementation, the merging module is used to calculate the color histogram or directional gradient histogram of any target image block based on the pixel information of each pixel in the target image block, thereby obtaining the image features of the target image block.

[0025] In one possible implementation, the device further includes:

[0026] The acquisition module is used to acquire image features from multiple sample image blocks; and to acquire the first target classification network based on the image features of each sample image block.

[0027] The merging module is used to input the image features of each target image block into the first target classification network to obtain the category of each target image block.

[0028] In one possible implementation, the acquisition module is configured to perform clustering processing on the plurality of sample image blocks according to the image features of each sample image block to obtain a first category of each sample image block; input the image features of each sample image block into a first initial classification network to obtain a second category of each sample image block; and adjust the first initial classification network according to the first category and the second category of each sample image block to obtain a first target classification network.

[0029] In one possible implementation, the merging module is used to perform clustering processing on the multiple target image blocks according to the image features of each target image block to obtain the category of each target image block.

[0030] In one possible implementation, the merging module is configured to obtain clustering indices for the number of multiple candidate categories, wherein the clustering indices for any one candidate category are used to characterize the accuracy of clustering the multiple target image patches into the number of categories for any one candidate category; select the number of target categories corresponding to the largest clustering index from the multiple candidate category numbers; and cluster the multiple target image patches into the number of target categories according to the image features of each target image patch to obtain the category of each target image patch.

[0031] In one possible implementation, the merging module is configured to determine the position of each target image block in the target image and the category of each target image block; for any two target image blocks, if the two target image blocks are of the same category and are adjacent in the target image, then the two target image blocks are merged into one target image region; if the two target image blocks are of different categories, or are not adjacent in the target image, or are of different categories and are not adjacent in the target image, then the two target image blocks are merged into two target image regions.

[0032] In one possible implementation, the determining module is configured to, for any target image region, acquire at least one of the image features, category, and size information of each target image block in the target image region; and determine the image features of the target image region based on at least one of the image features, category, and size information of each target image block in the target image region.

[0033] In one possible implementation, the determining module is configured to determine a graph structure based on the image features of each target image region and the position of each target image region in the target image. The graph structure includes at least two nodes and at least one edge. A node represents the image feature of a target image region, and an edge represents the positional adjacency of the target image regions corresponding to the two nodes at both ends of the edge. The module is then used to determine the category of the target image based on the graph structure.

[0034] In one possible implementation, the device further includes:

[0035] The merging module is further configured to merge the multiple sample image blocks into at least two sample image regions based on the correlation between the multiple sample image blocks, wherein the sample image region includes at least one sample image block;

[0036] The determining module is also used to determine the image features of each sample image region;

[0037] The acquisition module is used to acquire the second target classification network based on the image features of each sample image region;

[0038] The determining module is used to input the image features of each target image region into the second target classification network to obtain the category of the target image.

[0039] In one possible implementation, the acquisition module is used to acquire the labeled category of the sample image; input the image features of each sample image region into a second initial classification network to obtain the predicted category of the sample image; and adjust the second initial classification network based on the labeled category and the predicted category of the sample image to obtain the second target classification network.

[0040] On the other hand, embodiments of this application provide an electronic device, which includes a processor and a memory. The memory stores at least one computer program, which is loaded and executed by the processor to enable the electronic device to implement any of the image classification methods described above.

[0041] On the other hand, a computer-readable storage medium is also provided, wherein at least one computer program is stored therein, the at least one computer program being loaded and executed by a processor to enable a computer to implement any of the image classification methods described above.

[0042] On the other hand, a computer program or computer program product is also provided, wherein at least one computer program is stored in the computer program or computer program product, and the at least one computer program is loaded and executed by a processor to enable the computer to implement any of the above-mentioned image classification methods.

[0043] The technical solution provided in this application has at least the following beneficial effects:

[0044] The technical solution provided in this application, after segmenting a target image into multiple target image blocks, merges these blocks into at least two target image regions based on the correlation between them, and determines the category of the target image based on the image features of each target image region. This achieves the goal of determining the category of the target image by comprehensively considering the correlation between target image blocks, thus improving the accuracy of image classification results. Furthermore, merging multiple target image blocks into at least two target image regions reduces the number of blocks and improves the efficiency of image classification processing. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a schematic diagram of the implementation environment of an image classification method provided in an embodiment of this application;

[0047] Figure 2 This is a flowchart of an image classification method provided in an embodiment of this application;

[0048] Figure 3 This is a schematic diagram of an image segmentation method provided in an embodiment of this application;

[0049] Figure 4 This is a schematic diagram of the training of an initial feature extraction network provided in an embodiment of this application;

[0050] Figure 5 This is a training diagram of a first initial classification network provided in an embodiment of this application;

[0051] Figure 6 This is a schematic diagram illustrating the construction of a graph structure provided in an embodiment of this application;

[0052] Figure 7 This is a training diagram of a second initial classification network provided in an embodiment of this application;

[0053] Figure 8 This is a schematic diagram illustrating a method for training an image classification model using pathological images, as provided in an embodiment of this application.

[0054] Figure 9 This is a schematic diagram of the structure of an image classification device provided in an embodiment of this application;

[0055] Figure 10 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application;

[0056] Figure 11 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0058] Figure 1 This is a schematic diagram illustrating the implementation environment of an image classification method provided in this application embodiment, such as... Figure 1As shown, the implementation environment includes a terminal device 101 and a server 102. The image classification method in this embodiment can be executed by the terminal device 101, by the server 102, or by both the terminal device 101 and the server 102.

[0059] Terminal device 101 can be a smartphone, game console, desktop computer, tablet computer, laptop computer, smart TV, smart in-vehicle device, smart voice interaction device, smart home appliance, etc. Server 102 can be a single server, a server cluster consisting of multiple servers, or any of the following: cloud computing platform and virtualization center. This application embodiment does not limit this. Server 102 can communicate with terminal device 101 via a wired network or wireless network. Server 102 can have functions such as data processing, data storage, and data transmission and reception. This application embodiment does not limit this. The number of terminal devices 101 and servers 102 is not limited and can be one or more.

[0060] The image classification method in this application embodiment can be implemented using artificial intelligence (AI) technology. Artificial intelligence is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, artificial intelligence is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can react in a way similar to human intelligence. Artificial intelligence also studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making functions.

[0061] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and intelligent transportation.

[0062] Computer vision (CV) is the science that studies how to enable machines to "see." More specifically, it refers to machine vision, which uses cameras and computers to replace human eyes in recognizing and measuring targets, and then performs image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, attempting to build artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), autonomous driving, and intelligent transportation technologies.

[0063] Based on the above implementation environment, this application provides an image classification method to... Figure 2 The flowchart shown in this embodiment of the application illustrates an image classification method. This method can be implemented by... Figure 1 The method can be executed by either terminal device 101 or server 102, or jointly by both. For ease of description, the terminal device 101 or server 102 executing the image classification method in this embodiment is referred to as an electronic device, and the method can be executed by an electronic device. Figure 2 As shown, the method includes steps 201 to 204.

[0064] Step 201: Segment the target image to be classified into multiple target image blocks.

[0065] This application does not limit the image type, image size, etc., of the target image to be classified. For example, the target image is a digital image obtained by scanning a pathological slide using a digital pathology slide scanner (composed of an optical system, a linear scanning camera, etc.). In other words, the target image can be a pathological image. A pathological image can be called a whole-slide image (WSI), and typically, the size of a WSI is greater than or equal to 10,000 pixels by 10,000 pixels. The target image can also be a remote sensing image; the size of remote sensing images is not limited here.

[0066] The number of target images is at least one. For each target image, the category of the target image can be determined in accordance with steps 201 to 204.

[0067] The target image can be segmented to obtain multiple target image blocks of the same size or resolution. Optionally, the target image can be segmented into target image blocks of the same size as the window using a sliding window method. The window size and resolution can be set arbitrarily. For example, a resolution of 224 pixels * 224 pixels under 20x magnification means that each target image block has a resolution of 224 pixels * 224 pixels under 20x magnification. When the target image is a pathological image, the target image blocks can be called pathological image blocks; when the target image is a remote sensing image, the target image blocks can be called remote sensing image blocks.

[0068] Please see Figure 3 , Figure 3 This is a schematic diagram of image segmentation provided in an embodiment of this application. Wherein, Figure 3 (1) is the target image, and the dashed lines on the target image represent the segmentation lines. The target image is divided into multiple target image blocks of the same size according to the segmentation lines on the target image. Figure 3 (2) shows 12 target image blocks of the same size.

[0069] Step 202: Based on the correlation between multiple target image blocks, merge the multiple target image blocks into at least two target image regions, each target image region including at least one target image block.

[0070] This application does not limit the association between target image blocks. For example, the association between target image blocks includes at least one of the following: association between target image blocks by category, association between target image blocks by position, association between target image blocks by pixel information, and association between target image blocks by image features. Through the association between multiple target image blocks, at least according to any one of implementations A1 to A6, multiple target image blocks can be merged into at least two target image regions. Wherein, when the target image is a pathological image, the target image region can be called a pathological image region; when the target image is a remote sensing image, the target image region can be called a remote sensing image region.

[0071] In implementation method A1, the association relationship between target image blocks includes the positional association relationship between target image blocks. Optionally, based on the association relationship between multiple target image blocks, multiple target image blocks are merged into at least two target image regions, including: determining the position of each target image block in the target image; merging multiple target image blocks into at least two target image regions based on the position and quantity threshold of each target image block in the target image, wherein the number of target image blocks included in the target image region is less than or equal to the quantity threshold, and any two target image blocks included in the target image region are adjacent in position in the target image.

[0072] In this embodiment, the position of each target image block in the target image can be determined. The position of the target image block includes, but is not limited to, the length, width, area, perimeter, center point coordinates, and at least one vertex coordinate of the target image block. Based on the position and quantity threshold of each target image block in the target image, multiple target image blocks can be merged into at least two target image regions. The value of the quantity threshold is not limited here, but the quantity threshold is a positive integer greater than 1, and the shape of the target image region is also not limited here.

[0073] by Figure 3 Taking (2) as an example, the quantity threshold value is 4. Based on the position of each target image block in the target image and the quantity threshold of 4, the target image blocks in the first row and first column, the first row and second column, the second row and first column, and the second row and second column are merged into one target image region. The target image blocks in the first row and third column, the first row and fourth column, the second row and third column, and the second row and fourth column are merged into one target image region. The target image blocks from the third row and first column to the third row and fourth column are merged into one target image region.

[0074] In implementation method A2, the association between target image blocks includes the association between their categories. Optionally, based on the association between multiple target image blocks, multiple target image blocks are merged into at least two target image regions, including: determining the category of each target image block; for any two target image blocks, if the categories of any two target image blocks are the same, then the two target image blocks are merged into one target image region; if the categories of any two target image blocks are different, then the two target image blocks are merged into two target image regions.

[0075] In this embodiment of the application, the category of each target image patch can be determined. The method for determining the category of the target image patch is described in detail below and will not be repeated here.

[0076] For any two target image patches, if the two target image patches are of the same category, then the two target image patches are merged into one target image region. Figure 3 For example, in (2), if the target image block in the first row and first column and the target image block in the third row and second column are of the same category, then the target image block in the first row and first column and the target image block in the third row and second column will be merged into one target image region.

[0077] If any two target image patches belong to different categories, then the two target image patches will be merged into two target image regions. Figure 3 For example, in (2), if the target image block in the first column of the third row and the target image block in the second column of the third row are of different categories, then the target image block in the first column of the third row and the target image block in the second column of the third row will be merged into two target image regions.

[0078] In implementation method A3, the association relationship between target image blocks includes the association relationship of pixel information between target image blocks. Optionally, based on the association relationship between multiple target image blocks, multiple target image blocks are merged into at least two target image regions, including: determining the average pixel information of any target image block according to the pixel information of each pixel in any target image block; for any two target image blocks, if the difference between the average pixel information of any two target image blocks is less than a first threshold, then the two target image blocks are merged into one target image region; if the difference between the average pixel information of any two target image blocks is not less than the first threshold, then the two target image blocks are merged into two target image regions. The pixel information of a pixel can be at least one of Red Green Blue (RGB) color information, grayscale value, brightness value, etc., and this application embodiment does not limit the detailed value of the first threshold.

[0079] In implementation method A4, the association between target image blocks includes the association between image features of the target image blocks. Optionally, based on the association between multiple target image blocks, multiple target image blocks are merged into at least two target image regions, including: obtaining the image features of each target image block; for any two target image blocks, determining the distance between the two target image blocks based on the image features of the two target image blocks; if the distance between any two target image blocks is less than a second threshold, then the two target image blocks are merged into one target image region; if the distance between any two target image blocks is not less than the second threshold, then the two target image blocks are merged into two target image regions.

[0080] The method for determining the image features of the target image patch is described in detail below and will not be repeated here.

[0081] In this embodiment, any distance algorithm can be used to determine the distance between any two target image patches based on their image features. Then, based on the distance between the two target image patches and a second threshold, it is determined whether to merge the two target image patches into one target image region or merge them into two target image regions. This embodiment does not limit the specific value of the second threshold.

[0082] In application, multiple target image blocks can be merged into at least two target image regions based on the implementation principles of at least two of the implementation methods A1 to A4. Since information from at least two dimensions is referenced when merging target image blocks, the accuracy of the merging process is improved, thereby increasing the accuracy of the target image regions. For example, based on the implementation principles of implementation methods A1 and A2, the principle of implementation method A5 is derived, and based on the principle of implementation method A5, multiple target image blocks are merged into at least two target image regions.

[0083] In implementation method A5, the association between target image blocks includes the association between categories and the association between positions of target image blocks. Optionally, based on the association between multiple target image blocks, multiple target image blocks are merged into at least two target image regions, including: determining the position of each target image block in the target image and the category of each target image block; for any two target image blocks, if any two target image blocks have the same category and are adjacent in the target image, then the two target image blocks are merged into one target image region; if any two target image blocks have different categories, or are not adjacent in the target image, or have different categories and are not adjacent in the target image, then the two target image blocks are merged into two target image regions.

[0084] In this embodiment of the application, for any two target image blocks, if the two target image blocks are of the same category and are adjacent in position within the target image, then the two target image blocks are merged into one target image region. Figure 3 For example, in (2), if the target image block in the first row and first column and the target image block in the first row and second column are of the same category, since the target image block in the first row and first column and the target image block in the first row and second column are adjacent in the target image, the target image block in the first row and first column and the target image block in the first row and second column are merged into one target image region.

[0085] If any two target image patches are of different categories, but are adjacent in position within the target image, then the two target image patches will be merged into two target image regions. Figure 3 For example, in (2), if the target image block in the first row and first column and the target image block in the first row and second column are of different categories, even if the target image block in the first row and first column and the target image block in the first row and second column are adjacent in the target image, the target image block in the first row and first column and the target image block in the first row and second column will still be merged into two target image regions.

[0086] If any two target image patches are not adjacent in location within the target image, but belong to the same category, then the two target image patches will be merged into two target image regions. Figure 3 For example, in (2), if the target image block in the first row and first column and the target image block in the first row and third column are of the same category, since the target image block in the first row and first column and the target image block in the first row and third column are not adjacent in the target image, the target image block in the first row and first column and the target image block in the first row and third column are merged into two target image regions.

[0087] If any two target image patches are of different categories and are not adjacent in the target image, then the two target image patches will be merged into two target image regions. Figure 3 For example, in (2), if the target image block in the first row and first column and the target image block in the first row and third column are of different categories, and since the target image block in the first row and first column and the target image block in the first row and third column are not adjacent in the target image, the target image block in the first row and first column and the target image block in the first row and third column are merged into two target image regions.

[0088] Optionally, after obtaining each target image region through implementation method A5, if the number of target image blocks in any target image region is greater than the number threshold, then the target image region is split into at least two target image regions so that the number of target image blocks in each split target image region is less than or equal to the number threshold.

[0089] For example, based on the implementation principles of implementation methods A1 to A3, the implementation principle of implementation method A6 is obtained. Based on the principle of implementation method A6, multiple target image blocks are merged into at least two target image regions.

[0090] In implementation method A6, the association relationships between target image blocks include the association relationships between target image blocks by category, the association relationships between target image blocks by position, and the association relationships between target image blocks by pixel information. Optionally, based on the association relationships between multiple target image blocks, multiple target image blocks are merged into at least two target image regions, including: determining the position and category of each target image block in the target image; determining the average pixel information of any target image block based on the pixel information of each pixel in any target image block; for any two target image blocks, if the two target image blocks have the same category, are adjacent in the target image, and the difference between the average pixel information of the two target image blocks is less than a first threshold, then the two target image blocks are merged into one target image region; if at least one of the following is satisfied: the two target image blocks have different categories, are not adjacent in the target image, and the difference between the average pixel information of the two target image blocks is not less than a first threshold, then the two target image blocks are merged into two target image regions.

[0091] Optionally, after obtaining each target image region through implementation method A6, if the number of target image blocks in any target image region is greater than the number threshold, then the target image region is split into at least two target image regions so that the number of target image blocks in each split target image region is less than or equal to the number threshold.

[0092] In this embodiment of the application, based on the implementation principles of at least two of the implementation methods A1 to A4, the method of merging multiple target image blocks into at least two target image regions can be seen in implementation methods A5 and A6, and their implementation principles are similar, so they will not be described again here.

[0093] It should be noted that, since the texture information of identical content in the target image is similar, the positions of identical content are adjacent, and the categories, pixel information, and image features of identical content are also similar. Based on at least one of implementation methods A1 to A4, multiple target image blocks are merged into at least two target image regions, which can merge identical content in the target image together, thereby achieving accurate segmentation of the target image.

[0094] Taking a pathological image as an example, the above method segments the pathological image into multiple pathological image blocks. Since the texture information of the same tissue structures in a pathological image is similar, the positions of the same tissue structures are adjacent, and the categories, pixel information, and image features of the same tissue structures are also similar. Based on at least one of the following relationships: the correlation between categories, the correlation between positions, the correlation between pixel information, and the correlation between image features among multiple pathological image blocks, multiple pathological image blocks are merged into at least two pathological image regions to combine the same tissue structures together, thereby achieving accurate segmentation of the pathological image.

[0095] When merging multiple target image blocks into at least two target image regions based on the implementation principle of implementation method A2, it is necessary to determine the category of each target image block.

[0096] In this embodiment, the category of each target image patch can be determined in various ways. For example, a clustering algorithm can be used to cluster multiple target image patches to obtain the category of each target image patch. In this case, the category of any target image patch is the cluster to which that target image patch belongs. Alternatively, a first target classification network can be used to classify each target image patch to obtain the category of each target image patch.

[0097] Optionally, determining the category of each target image patch includes: acquiring the image features of each target image patch, wherein the image features of any target image patch are used to describe the texture information of any target image patch; and determining the category of each target image patch based on the image features of each target image patch.

[0098] For any target image block, the image features of the target image block can be determined based on the pixel information of each pixel in the target image block, so as to use the image features of the target image block to describe the texture information of the target image block.

[0099] In one possible implementation, before obtaining the image features of each target image block, the method further includes: segmenting the sample image into multiple sample image blocks; obtaining a target feature extraction network based on the multiple sample image blocks; and obtaining the image features of each target image block, including: obtaining the image features of each target image block based on the target feature extraction network.

[0100] In this embodiment, a sample image can be acquired and segmented to obtain multiple sample image blocks of the same size or resolution. The segmentation method for the sample image is similar to that for the target image and will not be described again here. The sample image and the target image have the same image type; that is, the sample image can be a pathological image or a remote sensing image. When the sample image is a pathological image, the sample image block is a pathological image block; when the sample image is a remote sensing image, the sample image block is a remote sensing image block.

[0101] After obtaining multiple sample image patches, a target feature extraction network can be derived based on these patches. Optionally, the initial feature extraction network can be adjusted using a contrastive learning approach based on the multiple sample image patches to obtain the target feature extraction network.

[0102] Optionally, a target feature extraction network is obtained based on multiple sample image blocks, including: performing a first data augmentation process on each sample image block to obtain sample image blocks after the first data augmentation process; performing a second data augmentation process on each sample image block to obtain sample image blocks after the second data augmentation process; obtaining image features of each sample image block after the first data augmentation process and image features of each sample image block after the second data augmentation process based on an initial feature extraction network; and adjusting the initial feature extraction network based on the image features of each sample image block after the first data augmentation process and image features of each sample image block after the second data augmentation process to obtain a target feature extraction network.

[0103] In this embodiment of the application, for any sample image block, a first data augmentation process can be performed on the sample image block to obtain a sample image block after the first data augmentation process. Alternatively, a second data augmentation process can be performed on the sample image block to obtain a sample image block after the second data augmentation process.

[0104] The first data enhancement process includes geometric transformation, contrast adjustment, pixel perturbation, noise addition, illumination adjustment, and interpolation. Geometric transformation includes, but is not limited to, cropping, flipping, rotating, scaling, and distorting. The second data enhancement process is different from the first; for example, the first process might be contrast adjustment, and the second process might be flipping. Alternatively, the second and first processes might be the same, but with different magnitudes; for example, the first process might be rotation (90 degrees), and the second process might also be rotation (180 degrees).

[0105] In this embodiment, any sample image block after first data augmentation is input into an initial feature extraction network. The initial feature extraction network determines and outputs the image features (high-dimensional features) of the sample image block based on the pixel information of each pixel in the first data augmentation. Similarly, any sample image block after second data augmentation is input into the initial feature extraction network. The initial feature extraction network determines and outputs the image features (also high-dimensional features) of the sample image block based on the pixel information of each pixel in the second data augmentation. This embodiment does not limit the network structure or size of the initial feature extraction network. For example, the initial feature extraction network can be a residual network (such as ResNet18), a VisualGeometry Group (VGG) network, or a Google Net. This embodiment does not limit the dimension of the high-dimensional features. For example, the dimension of the high-dimensional features is 512.

[0106] Next, using the image features of any sample image patch after the first data augmentation and the image features of any sample image patch after the second data augmentation, a first loss value for any sample image patch is determined according to a loss function. This loss function can be a contrastive loss function. In this way, the first loss value for each sample image patch can be determined. Then, based on the first loss value of each sample image patch, the loss value of the initial feature extraction network is determined, and the initial feature extraction network is adjusted using this loss value to obtain the adjusted initial feature extraction network.

[0107] When the first training termination condition is met, the adjusted initial feature extraction network becomes the target feature extraction network. When the first training termination condition is not met, the initial feature extraction network is adjusted again based on multiple sample image patches, following the above method, until the first training termination condition is met. The adjusted initial feature extraction network then becomes the target feature extraction network. This embodiment does not limit the scope of meeting the first training termination condition. For example, meeting the first training termination condition means reaching the first training iteration, or it means the gradient of the loss value of the initial feature extraction network no longer decreases.

[0108] Please see Figure 4 , Figure 4 This is a schematic diagram illustrating the training of an initial feature extraction network according to an embodiment of this application. When training the initial feature extraction network, sample image patches are first acquired. On one hand, the sample image patches undergo a first data augmentation process to obtain augmented sample image patches. These augmented sample image patches are then input into the initial feature extraction network, which determines and outputs the image features of the augmented sample image patches. On the other hand, the sample image patches undergo a second data augmentation process to obtain augmented sample image patches. These augmented sample image patches are then input into the initial feature extraction network, which determines and outputs the image features of the augmented sample image patches. Based on the image features of the augmented sample image patches and the second augmented sample image patches, the training of the initial feature extraction network is supervised using a contrastive learning loss function to obtain the target feature extraction network.

[0109] After obtaining the target feature extraction network using the above method, for any target image patch, this patch is input into the target feature extraction network. The network then determines and outputs the high-dimensional features of the target image patch based on the pixel information of each pixel. These high-dimensional features are then used as the image features of the target image patch. In this way, the image features of each target image patch can be obtained.

[0110] The target feature extraction network determines the high-dimensional features of the target image patch as follows: For any pixel in any target image patch, the target feature extraction network updates the features of that pixel based on its pixel information and the pixel information of all surrounding pixels. Based on the features of each pixel in the target image patch, the target feature extraction network determines and outputs the high-dimensional features of that target image patch.

[0111] In this embodiment, the target feature extraction network is trained based on multiple sample image patches, and no annotation is required for any of the sample image patches before or during training. In other words, the target feature extraction network is trained without any annotations. Since no annotations are required, the consumption of human and material resources can be reduced, and training efficiency can be improved.

[0112] In one possible implementation, the image features of each target image patch are obtained, including:

[0113] For any target image patch, calculate the color histogram or orientation gradient histogram of the target image patch based on the pixel information of each pixel in the target image patch to obtain the image features of the target image patch.

[0114] In this embodiment, for any target image block, a color histogram can be calculated based on the pixel information of each pixel in the target image block. The color histogram describes the proportion of different colors in the entire image, without concern for the spatial position of each color. The color histogram of any target image block can be the global color histogram of the target image block, the accumulated color histogram of the target image block, or a histogram determined based on both the global and accumulated color histograms of the target image block.

[0115] In one possible implementation, a computer vision library (such as OpenCV) can be used to calculate the color histogram of any given target image patch based on the pixel information of each pixel. OpenCV is a cross-platform computer vision and machine learning software library that can run on various operating systems.

[0116] In this embodiment, for any target image block, its histogram of oriented gradient (HOG) can be calculated based on the pixel information of each pixel in the target image block. When calculating the HOG of any target image block, the target image block can first be normalized to make it a grayscale image, resulting in a normalized target image block. Then, the gradient of the normalized target image block is calculated using the first derivative. Finally, based on the gradient of the normalized target image block, the HOG of the target image block is determined.

[0117] Optionally, the image features of any target image patch can be determined based on at least one of the color histogram, the orientation gradient histogram, and the high-dimensional features of any target image patch. For example, the image features of any target image patch can be obtained by concatenating the color histogram, the orientation gradient histogram, and the high-dimensional features of any target image patch.

[0118] In one possible implementation, before determining the category of each target image patch based on its image features, the method further includes: acquiring image features of multiple sample image patches; acquiring a first target classification network based on the image features of each sample image patch; and determining the category of each target image patch based on its image features, including: inputting the image features of each target image patch into the first target classification network to obtain the category of each target image patch.

[0119] For any given sample image patch, it can be input into the target feature extraction network. The network then determines and outputs the high-dimensional features of each sample image patch based on the pixel information of each pixel. These high-dimensional features are then used as the image features of the sample image patch. Alternatively, the color histogram or directional gradient histogram of the sample image patch can be calculated based on the pixel information of each pixel to obtain its image features. Furthermore, the image features of any sample image patch can be determined based on at least two of the high-dimensional features, color histograms, or gradient histograms. The methods for determining the image features of sample image patches are similar to those for determining the image features of target image patches, and will not be elaborated further here.

[0120] After obtaining the image features of each sample image block, the first initial classification network is adjusted according to the image features of each sample image block to obtain the first target classification network, so as to determine the category of the target image block.

[0121] Optionally, obtaining the first target classification network based on the image features of each sample image block includes: performing clustering processing on multiple sample image blocks based on the image features of each sample image block to obtain a first category of each sample image block; inputting the image features of each sample image block into a first initial classification network to obtain a second category of each sample image block; and adjusting the first initial classification network based on the first category and the second category of each sample image block to obtain the first target classification network.

[0122] Any clustering algorithm can be used to cluster multiple sample image patches based on their image features, resulting in a first category for each sample image patch. Optional clustering algorithms include, but are not limited to, K-means clustering, spectral clustering, and Expectation Maximization (EM) clustering based on a Gaussian Mixed Model (GMM). The use of any clustering algorithm to cluster multiple sample image patches based on their image features is described below in the section on "Clustering Multiple Target Image Patches Based on Their Image Features Using Any Clustering Algorithm," and will not be repeated here.

[0123] In this embodiment, the image features of each sample image block can be input into a first initial classification network, which then determines and outputs the second category of each sample image block. This embodiment does not limit the network structure or size of the first initial classification network; for example, the first initial classification network can be a neural network or a Support Vector Machine (SVM) network.

[0124] For any given sample image patch, a second loss value can be determined based on the first category and the second category of that sample image patch. Based on the second loss values ​​of each sample image patch, the loss value of the first initial classification network is determined, and the first initial classification network is adjusted using these loss values ​​to obtain the adjusted first initial classification network.

[0125] When the second training termination condition is met, the adjusted first initial classification network becomes the first target classification network. When the second training termination condition is not met, the first initial classification network is adjusted again based on multiple sample image patches in the manner described above until the second training termination condition is met, and the adjusted first initial classification network becomes the first target classification network. This embodiment does not limit the scope of meeting the second training termination condition. For example, meeting the second training termination condition means reaching the second training iteration, or it means the gradient of the loss value of the first initial classification network no longer decreases.

[0126] Please see Figure 5 , Figure 5This is a schematic diagram illustrating the training of a first initial classification network according to an embodiment of this application. In this embodiment, multiple sample image blocks are input into a target feature extraction network, which determines and outputs the image features of each sample image block. Then, based on the image features of each sample image block, the multiple sample images are clustered to obtain at least two clusters. Each cluster includes at least one sample image block, and the cluster to which any sample image block belongs is the first category of that sample image block. The first initial classification network can be trained based on at least two clusters to obtain a first target classification network.

[0127] The first target classification network can be obtained through the above method. Next, the image features of each target image patch are input into the first target classification network, which then determines and outputs the category of each target image patch. Specifically, the first target classification network determines the category of the target image patch as follows: For any target image patch, the first target classification network performs a linear transformation on the image features of that target image patch to map the image features of that target image patch onto the category distribution dimension, obtaining the probability that any target image patch belongs to each category, and determining the category corresponding to the highest probability as the category of that target image patch.

[0128] In this embodiment, the first target classification network is trained based on the image features of multiple sample image blocks, and no annotation is required for any of the sample image blocks before or during training. In other words, the first target classification network is trained without any annotations. Since no annotations are required, the consumption of human and material resources can be reduced, and training efficiency can be improved.

[0129] In one possible implementation, the category of each target image block is determined based on its image features, including: clustering multiple target image blocks according to their image features to obtain the category of each target image block.

[0130] Any clustering algorithm can be used to cluster multiple target image patches based on their image features, resulting in a first category for each target image patch. Optional clustering algorithms include, but are not limited to, K-means, spectral clustering, and EM clustering based on GMM. The following section uses K-means as an example to detail the method of clustering multiple target image patches based on their image features.

[0131] First, determine the number of target categories, i.e., the desired number of clusters for multiple target image patches, denoted as k (k is a positive integer). Randomly select k image features from the image features of the multiple target image patches; each of these k target image patches can serve as a cluster center. For any given target image patch, using its image features and the image features of each cluster center, determine the distances between that target image patch and each cluster center. Cluster the target image patch with the cluster center that has the smallest distance into the same category. In this way, multiple target image patches can be clustered into k categories.

[0132] Next, for each category, the image features of the cluster centers corresponding to that category are updated using the image features of each target image patch within that category, resulting in the updated image features of the cluster centers for that category. For each category, the distance between the unupdated and updated cluster centers corresponding to that category is determined using both the unupdated and updated image features of the cluster centers.

[0133] If the distance between the pre-updated and post-updated cluster centers for each category is less than the distance threshold, the clustering result is considered stable, and iteration can be stopped, resulting in k categories. If at least one category has a distance between the pre-updated and post-updated cluster centers that is not less than the distance threshold, then the post-updated cluster centers for each category are used as the respective cluster centers. Iteration continues from "For any target image patch among multiple target image patches, using the image features of the target image patch and the image features of each cluster center, determine the distance between the target image patch and each cluster center, and cluster the target image patch and the cluster center with the smallest distance into the same category," until the distance between the pre-updated and post-updated cluster centers for each category is less than the distance threshold, resulting in k categories.

[0134] Using the method described above, multiple target image patches can be clustered into k categories, with each category including at least one target image patch. In other words, the category of each target image patch can be obtained through this method.

[0135] In this embodiment, using any clustering algorithm, after clustering multiple target image blocks according to their image features, at least two clusters can be obtained. Each cluster includes at least one target image block, and the cluster to which a target image block belongs is the category of that target image block.

[0136] Optionally, based on the image features of each target image patch, clustering is performed on multiple target image patches to obtain the category of each target image patch. This includes: obtaining a clustering index for the number of multiple candidate categories, where the clustering index for any candidate category is used to characterize the accuracy of clustering multiple target image patches into any number of candidate categories; selecting the number of target categories corresponding to the largest clustering index from the multiple candidate categories; and clustering multiple target image patches into the number of target categories based on the image features of each target image patch to obtain the category of each target image patch.

[0137] In this embodiment of the application, the number of multiple candidate categories can be determined. These multiple candidate category numbers can be consecutive, such as determining 2-60 as multiple candidate category numbers, or they can be non-consecutive, such as determining 4, 9, and 16 as multiple candidate category numbers.

[0138] In this embodiment, for each number of candidate categories, any clustering algorithm can be used to cluster multiple target image blocks into that number of candidate categories based on the image features of each target image block. For any given target image block, on one hand, the distance between the given target image block and each target image block in its category is determined based on the image features of the given target image block and the image features of each target image block in its category. Based on the distance between the given target image block and each target image block in its category, a first average distance is determined for that given target image block. On the other hand, the image features of the cluster centers of the category of the given target image block are determined, and the image features of the cluster centers of all other categories besides the category of the given target image block are determined. Based on the image features of the cluster centers of the category of the given target image block and the image features of the cluster centers of all other categories, the distance between the cluster centers of the category of the given target image block and the cluster centers of all other categories is determined, and the other categories with the smallest distance are determined. Based on the image features of any target image patch and the image features of each target image patch in the other categories with the minimum distance, the distance between any target image patch and each target image patch in the other categories with the minimum distance is determined. Based on the distance between any target image patch and each target image patch in the other categories with the minimum distance, the second average distance corresponding to that target image patch is determined. Then, based on the first average distance and the second average distance corresponding to any target image patch, the silhouette coefficient (SC) corresponding to that target image patch is determined, as shown in the following formula (1).

[0139]

[0140] Where SC(i) represents the contour coefficient corresponding to the i-th target image patch, b(i) represents the second average distance corresponding to the i-th target image patch, a(i) represents the first average distance corresponding to the i-th target image patch, and max is the maximum value symbol.

[0141] The first average distance corresponding to the target image patch is used to reflect the distance between the target image patch and each target image patch in its category. The second average distance corresponding to the target image patch is used to reflect the distance between the target image patch and each target image patch in the other categories with the smallest distance, wherein the other categories with the smallest distance are the categories other than the target image patch's category among the number of candidate categories.

[0142] Next, the average contour coefficients corresponding to each target image patch are determined to obtain the contour coefficients for the number of candidate categories. The contour coefficients for the number of candidate categories characterize the accuracy of clustering multiple target image patches into that number of candidate categories; they can also be called clustering metrics for the number of candidate categories. Specifically, the contour coefficients for the number of candidate categories are directly proportional to the accuracy. That is, the larger the contour coefficients for the number of candidate categories, the higher the accuracy of clustering multiple target image patches into any given number of candidate categories; conversely, the smaller the contour coefficients for the number of candidate categories, the lower the accuracy of clustering multiple target image patches into any given number of candidate categories.

[0143] It should be noted that the silhouette coefficient for the number of candidate categories is a value greater than or equal to 0 and less than or equal to 1. The closer the silhouette coefficient for the number of candidate categories is to 1, the smaller the intra-class distance and the larger the out-of-class distance when clustering multiple target image patches into any number of candidate categories, and therefore the higher the accuracy.

[0144] Using the above method, we can obtain the silhouette coefficients for the number of candidate categories, which are essentially clustering indices for each candidate category. Next, we select the target category number corresponding to the largest clustering index from the multiple candidate category numbers. Since the target category number is just one of the multiple candidate category numbers, and the silhouette coefficients for determining the candidate category number have already clustered multiple target image patches into that number of candidate category numbers based on their image features, we can directly obtain the category of each target image patch by clustering multiple target image patches into that number of target category numbers based on their image features.

[0145] It should be noted that, in addition to selecting from multiple candidate categories, the number of target categories can also be determined based on experience, historical experimental data, etc.

[0146] Step 203: Determine the image features of each target image region.

[0147] In this embodiment of the application, for any target image region, the image features of that target image region can be determined based on the image features of each target image block within that region. For example, the image features of each target image block within a target image region can be combined into a matrix, where one row or one column of data in the matrix represents the image features of a target image block; this matrix represents the image features of any target image region. Alternatively, the average (or sum) of the image features of each target image block within a target image region can be calculated, and this average (or sum) can be used as the image features of the target image region.

[0148] Optionally, determining the image features of each target image region includes: for any target image region, acquiring at least one of the image features, category, and size information of each target image block in the target image region; and determining the image features of any target image region based on at least one of the image features, category, and size information of each target image block in the target image region.

[0149] The methods for obtaining the image features and categories of the target image blocks have already been described in step 202, and will not be repeated here. In this embodiment, the size information of each target image block in any target image region can also be obtained. The size information of the target image block includes at least one of the following: the center position of the target image block, the length of the target image block, the width of the target image block, the area of ​​the target image block, and the perimeter of the target image block.

[0150] Optionally, the image features of each target image patch within any target image region can be combined into a matrix, where one row or column of data in the matrix represents the image feature of a target image patch. This matrix can be used as the image feature of any target image region or as the first part of the feature set of any target image region. Alternatively, the average (or sum) of the image features of each target image patch within any target image region can be calculated, and this average (or sum) can be used as the image feature of any target image region or as the first part of the feature set of any target image region. The image features of the target image patch include at least one of the following: color histogram, oriented gradient histogram, or high-dimensional features output by the target feature extraction network.

[0151] The category of each target image patch within any given target image region can be determined. These target image patches may belong to the same or different categories. The category of each target image patch within a given target image region can be encoded (e.g., one-hot encoding, multi-hot encoding, etc.) to obtain the category code for each target image patch in the given target image region. These category codes are then combined into a matrix, where each row or column of data represents the category code of a single target image patch. This matrix serves as the image feature or the second part of the feature set for any given target image region. Alternatively, based on the category codes of each target image patch within a given target image region, an average value (or sum) can be calculated, and this average value (or sum) can be used as the image feature or the second part of the feature set for any given target image region.

[0152] The size information of each target image block in any target image region can be combined into a matrix. A row or column of data in the matrix represents the size information of a target image block. This matrix can be used as an image feature or a third-part feature of any target image region. Alternatively, based on the size information of each target image block in any target image region, the size information of any target image region can be calculated. This size information can be used as an image feature or a third-part feature of any target image region. The size information of any target image region includes at least one of the following: the center position of the target image region, the length of the target image region, the width of the target image region, the area of ​​the target image region, the perimeter of the target image region, the radius of the target image region, and the contour of the target image region.

[0153] In this embodiment of the application, at least two of the first part features, the second part features, and the third part features of any target image region are spliced ​​together to obtain the image features of any target image region.

[0154] The image features of each target image region can be determined using the methods described above.

[0155] Step 204: Determine the category of the target image based on the image features of each target image region.

[0156] In this embodiment, the target image is first segmented into multiple target image blocks, and then these blocks are merged into at least two target image regions, resulting in a target image comprising at least two target image regions. The target image can be classified based on the image features of each target image region to obtain its category. This embodiment does not limit the category of the target image. For example, if the target image is a pathological image, its category may include at least one category such as lesion type, tissue malignancy, severity of condition, or whether the condition has recurred. If the target image is a remote sensing image, its category may include categories such as whether the image contains a target object or the location of the remote sensing image.

[0157] Optionally, determining the category of the target image based on the image features of each target image region includes: determining a graph structure based on the image features of each target image region and the position of each target image region in the target image, wherein the graph structure includes at least two nodes and at least one edge, where a node represents the image feature of a target image region and an edge represents the positional adjacency of the target image regions corresponding to the two nodes at both ends of an edge; and determining the category of the target image based on the graph structure.

[0158] In this embodiment, the image features of each target image region are used as a node in a graph structure. That is, the number of nodes in the graph structure equals the number of target image regions. A graph structure is a data structure with nodes and edges, where edges connect two nodes. If any two target image regions are adjacent in position within the target image, an edge is added between the nodes corresponding to the image features of these two target image regions, thereby establishing a connection between nodes in the graph structure. Since the graph structure includes the image features of each target image region, and the edges in the graph structure establish the adjacent relationship between two target image regions, the graph structure can be used to represent the global information of the target image.

[0159] Please see Figure 6 , Figure 6This is a schematic diagram of the construction of a graph structure provided in an embodiment of this application. (1) shows the target image. Segmenting the target image blocks yields (2), which shows each target image block. (3) shows the first target classification network. Inputting each target image block into the first target classification network yields (4), which shows the category of each target image block. If any two target image blocks have different colors, it indicates that the categories of the two target image blocks are different; if any two target image blocks have the same color, it indicates that the categories of the two target image blocks are the same. Using the categories of each target image block, (5) is constructed, which shows the graph structure. If any two nodes in the graph structure have different colors, it indicates that the categories of the target image blocks in the target image regions corresponding to these two nodes are different; if any two nodes in the graph structure have the same color, it indicates that the categories of the target image blocks in the target image regions corresponding to these two nodes are the same.

[0160] Target images can be classified based on graph structures to obtain their categories. Since graph structures can represent global information about the target image, they can improve classification quality. Therefore, the category of the target image is obtained by comprehensively considering the global information of the target image, resulting in high accuracy.

[0161] Taking a pathological image as an example, this embodiment determines the graph structure of a pathological image based on the image features of each region and its location within the image. Since pathological image regions are regions of the same tissue structure, the graph structure can characterize the distribution and correlation relationships between different tissue structures, thus representing the global information of the pathological image. When determining the category of a pathological image based on its graph structure, the category is obtained by comprehensively considering the global information of the image, thereby improving the classification quality and accuracy of the pathological image.

[0162] Optionally, before determining the category of the target image based on the image features of each target image region, the method further includes: merging multiple sample image blocks into at least two sample image regions based on the correlation between them, each sample image region including at least one sample image block; determining the image features of each sample image region; obtaining a second target classification network based on the image features of each sample image region; and determining the category of the target image based on the image features of each target image region, including: inputting the image features of each target image region into the second target classification network to obtain the category of the target image. The first target classification network and the second target classification network can be the same network or different networks.

[0163] This application does not limit the association between sample image blocks. For example, the association between sample image blocks includes at least one of the following: association between categories, association between positions, association between pixel information, and association between image features. Multiple sample image blocks can be merged into at least two sample image regions based on the association between them. The method for merging multiple sample image blocks into at least two sample image regions is described in step 202; the implementation principles are similar and will not be repeated here. Specifically, when the sample image is a pathological image, the sample image region is a pathological image region; when the sample image is a remote sensing image, the sample image region is a remote sensing image region.

[0164] Subsequently, for any given sample image region, the image features of that region are determined based on the image features of each sample image block within that region. This allows us to determine the image features of each sample image region. The method for determining the image features of a sample image region is described in step 203; the underlying principles are similar and will not be repeated here.

[0165] After obtaining the image features of each sample image region, the second initial classification network can be adjusted based on the image features of each sample image region to obtain the second target classification network.

[0166] Optionally, obtaining a second target classification network based on the image features of each sample image region includes: obtaining the labeled category of the sample image; inputting the image features of each sample image region into a second initial classification network to obtain the predicted category of the sample image; and adjusting the second initial classification network based on the labeled category and the predicted category of the sample image to obtain the second target classification network.

[0167] In this embodiment of the application, sample images can be labeled to obtain the labeled categories of the sample images. Optionally, the categories of sample images can be manually labeled to obtain the labeled categories of the sample images.

[0168] Image features from multiple sample image regions can be input into a second initial classification network, which then determines and outputs the predicted category of the sample image. Optionally, a graph structure can be determined first based on the image features of each sample image region and the position of each sample image region within the sample image. This graph structure is then input into the second initial classification network, which determines and outputs the predicted category of the sample image. The graph structure includes at least two nodes and at least one edge. A node represents the image feature of a sample image region, and an edge indicates that the sample image regions corresponding to the two nodes at its ends are adjacent.

[0169] It should be noted that the graph structure determined based on the image features of each target image region and the position of each target image region within the target image can be called the target graph structure, while the graph structure determined based on the image features of each sample image region and the position of each sample image region within the sample image can be called the sample graph structure. The method for determining the sample graph structure can be found in the relevant description of determining the target graph structure in step 204; the implementation principles of both are similar and will not be repeated here.

[0170] In this embodiment, the loss value of the second initial classification network can be determined based on the labeled category and the predicted category of the sample image. The second initial classification network is then adjusted based on its loss value to obtain an adjusted second initial classification network.

[0171] When the third training termination condition is met, the adjusted second initial classification network becomes the second target classification network. If the third training termination condition is not met, the second initial classification network is adjusted again based on the image features of multiple sample image regions, following the above method, until the third training termination condition is met. The adjusted second initial classification network then becomes the second target classification network. This embodiment does not limit the scope of meeting the third training termination condition. For example, meeting the third training termination condition means reaching the third training iteration, or it means the gradient of the loss value of the second initial classification network no longer decreases.

[0172] It should be noted that the number of sample images is at least one. For each sample image, the graph structure of the sample image can be determined based on the image features of each region within the sample image and the position of each region within the sample image. Using the graph structure and labeled categories of each sample image, the second initial classification network is trained to obtain the second target classification network. Training the second target classification network using the graph structure of the sample images allows it to effectively extract global information from the image and classify the image based on this global information, thus improving the accuracy of the image classification results.

[0173] When a second target classification network is trained using the graph structure of sample images, the second initial classification network can optionally be a Graph Neural Network (GNN). A GNN is a computational network composed of at least one of graph convolution operations, attention mechanisms, etc., and can be regarded as a special type of Convolutional Neural Network (CNN). A CNN is a computational network composed of multiple convolution operations and is commonly used in deep learning and neural network learning.

[0174] Please see Figure 7 , Figure 7 This is a training schematic diagram of a second initial classification network provided in an embodiment of this application. In this embodiment, the graph structure of each sample image is first obtained, and then the graph structure of each sample image is input into the second initial classification network. The second initial classification network determines and outputs the predicted category of each sample image. Alternatively, the labeled category of each sample image can be obtained. Using the predicted category and the labeled category of each sample image, the second initial classification network is adjusted to obtain the second target classification network.

[0175] After obtaining the second target classification network, the image features of each target image region can be input into the second target classification network, which then determines and outputs the category of the target image. Optionally, a graph structure can be determined first based on the image features of each target image region and the position of each target image region within the target image, and then the graph structure can be input into the second target classification network, which then determines and outputs the category of the target image.

[0176] Optionally, the second target classification network determines the category of the target image in the following manner: the second target classification network fuses the image features of each target image region to obtain the image features of the target image, performs a linear transformation on the image features of the target image to map the image features of the target image onto the category distribution dimension, obtains the probability that the target image belongs to each category, and determines the category corresponding to the highest probability as the category of the target image.

[0177] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the target images and sample images involved in this application were obtained with full authorization.

[0178] The method described above, after segmenting the target image into multiple target image patches, merges these patches into at least two target image regions based on the correlation between them. Then, it determines the category of the target image based on the image features of each target image region. This achieves the goal of determining the target image category by comprehensively considering the correlation between target image patches, thus improving the accuracy of image classification results. Furthermore, merging multiple target image patches into at least two target image regions reduces the number of regions required and improves the efficiency of image classification processing.

[0179] Taking a pathological image as an example, the above method segments the pathological image into multiple pathological image blocks. Based on the correlation between these blocks, they are merged into at least two pathological image regions. This merges similar tissue structures within the pathological image, achieving accurate segmentation. Furthermore, determining the category of the pathological image based on the image features of each region improves the accuracy of image classification. Merging multiple blocks into at least two regions reduces the number of blocks and increases the efficiency of image classification.

[0180] The above describes the image classification method from the perspective of methodological steps. The image processing method of this application embodiment will now be described in detail using a specific scenario. In the scenario of this application embodiment, the sample image is a pathological image. The pathological image can be used to train a neural network model to obtain an image classification model. The neural network model includes an initial feature extraction network, a first initial classification network, and a second initial classification network. The image classification model includes a target feature extraction network, a first target classification network, and a second target classification network.

[0181] Please see Figure 8 , Figure 8 This is a schematic diagram of training an image classification model using pathological images according to an embodiment of this application. The training process includes steps 801 to 807.

[0182] Step 801: Segment the pathological image into multiple pathological image blocks. Optionally, using a sliding window method, the pathological image is segmented into individual pathological image blocks of the same size as the window, which are equivalent to the sample image blocks mentioned above.

[0183] Step 802: Train the initial feature extraction network using each pathological image patch to obtain the target feature extraction network. Optionally, the initial feature extraction network can be trained and adjusted using multiple pathological image patches in a contrastive learning manner to obtain the target feature extraction network.

[0184] Step 803: Obtain image features for each pathological image block using a target feature extraction network. Specifically, each pathological image block is input into the target feature extraction network, which then determines and outputs the image features of each pathological image block.

[0185] Step 804: Based on the image features of each pathological image block, cluster the multiple pathological image blocks to obtain the first category of each pathological image block. Any clustering algorithm can be used to cluster the multiple pathological image blocks based on their image features to obtain the first category of each pathological image block.

[0186] Step 805: Train the first initial classification network using the first category of each pathological image patch to obtain the first target classification network. Optionally, the image features of each pathological image patch can be input into the first initial classification network, which determines and outputs the second category of each pathological image patch. The first initial classification network can then be trained using the first and second categories of each pathological image patch to obtain the first target classification network.

[0187] Step 806: Determine the third category of each pathological image patch using the first target classification network, and construct a graph structure using the third category of each pathological image patch. Optionally, based on the third category of each pathological image patch and its position in the pathological image, merge multiple pathological image patches into at least two pathological image regions, determine the image features of each pathological image region, and determine the graph structure based on the image features of each pathological image region and its position in the pathological image. Here, the pathological image region corresponds to the sample image region mentioned above.

[0188] Step 807: Based on the graph structure and the labeled categories of the pathological images, train the second initial classification network to obtain the second target classification network. Optionally, input the graph structure into the second initial classification network, which determines and outputs the predicted category of the pathological image, obtains the labeled category of the manually annotated pathological image, and trains the second initial classification network using the predicted category and the labeled category of the pathological image to obtain the second target classification network.

[0189] According to steps 801 to 807, a target feature extraction network, a first target classification network, and a second target classification network can be trained. In other words, according to steps 801 to 807, an image classification model can be trained. Then, the image classification model can be used to determine the category of any target image. The target image is also a pathological image.

[0190] When using an image classification model to determine the category of a pathological image (i.e., the target image mentioned above), the pathological image can first be segmented into multiple pathological image patches (i.e., the target image patches mentioned above). Each pathological image patch is then input into a target feature extraction network, which determines and outputs the image features of each patch. Next, the image features of each pathological image patch are input into a first target classification network, which determines and outputs the category of each patch. Following this, based on the category and location of each patch within the pathological image, the multiple patches are merged into at least two pathological image regions (i.e., the target image regions mentioned above). Then, the image features of each pathological image region are determined. Based on these features and their location within the pathological image, a graph structure is determined, and this graph structure is input into a second target classification network, which determines and outputs the category of the pathological image.

[0191] This method, after segmenting a pathological image into multiple pathological image blocks, merges these blocks into at least two pathological image regions based on their category and location within the overall image. The category of the pathological image is then determined based on the image features of each region. This approach integrates the relationships between the categories of pathological image blocks and their locations within the image to determine the category of the pathological image, thus improving the accuracy of image classification results. Furthermore, merging multiple pathological image blocks into at least two regions reduces the number of blocks required and improves the efficiency of image classification processing.

[0192] Figure 9 The diagram shown is a structural schematic of an image classification device provided in an embodiment of this application. Figure 9 As shown, the device includes:

[0193] The segmentation module 901 is used to segment the target image to be classified into multiple target image blocks;

[0194] The merging module 902 is used to merge multiple target image blocks into at least two target image regions based on the correlation between multiple target image blocks, wherein each target image region includes at least one target image block.

[0195] The determination module 903 is used to determine the image features of each target image region;

[0196] The determination module 903 is also used to determine the category of the target image based on the image features of each target image region.

[0197] In one possible implementation, the merging module 902 is used to determine the position of each target image block in the target image; based on the position and quantity threshold of each target image block in the target image, the plurality of target image blocks are merged into at least two target image regions, wherein the number of target image blocks included in the target image region is less than or equal to the quantity threshold, and any two target image blocks included in the target image region are adjacent in the target image.

[0198] In one possible implementation, the merging module 902 is used to determine the category of each target image block; for any two target image blocks, if the categories of any two target image blocks are the same, then the two target image blocks are merged into one target image region; if the categories of any two target image blocks are different, then the two target image blocks are merged into two target image regions.

[0199] In one possible implementation, the merging module 902 is used to obtain the image features of each target image block, whereby the image features of any target image block are used to describe the texture information of any target image block; and the category of each target image block is determined based on the image features of each target image block.

[0200] In one possible implementation, the device further includes:

[0201] The segmentation module 901 is also used to segment the sample image into multiple sample image blocks;

[0202] The acquisition module is used to acquire the target feature extraction network based on multiple sample image patches;

[0203] The merging module 902 is used to obtain the image features of each target image block based on the target feature extraction network.

[0204] In one possible implementation, an acquisition module is used to perform a first data augmentation process on each sample image block to obtain each sample image block after the first data augmentation process; perform a second data augmentation process on each sample image block to obtain each sample image block after the second data augmentation process; acquire image features of each sample image block after the first data augmentation process and image features of each sample image block after the second data augmentation process based on an initial feature extraction network; and adjust the initial feature extraction network based on the image features of each sample image block after the first data augmentation process and image features of each sample image block after the second data augmentation process to obtain a target feature extraction network.

[0205] In one possible implementation, the merging module 902 is used to calculate the color histogram or orientation gradient histogram of any target image block based on the pixel information of each pixel in the target image block, so as to obtain the image features of the target image block.

[0206] In one possible implementation, the device further includes:

[0207] The acquisition module is used to acquire image features from multiple sample image blocks; and to acquire the first target classification network based on the image features of each sample image block.

[0208] The merging module 902 is used to input the image features of each target image block into the first target classification network to obtain the category of each target image block.

[0209] In one possible implementation, the acquisition module is used to perform clustering processing on multiple sample image blocks according to the image features of each sample image block to obtain a first category of each sample image block; input the image features of each sample image block into a first initial classification network to obtain a second category of each sample image block; and adjust the first initial classification network according to the first category and the second category of each sample image block to obtain a first target classification network.

[0210] In one possible implementation, the merging module 902 is used to perform clustering processing on multiple target image blocks based on the image features of each target image block to obtain the category of each target image block.

[0211] In one possible implementation, the merging module 902 is used to obtain a clustering index for the number of multiple candidate categories, wherein the clustering index for the number of any candidate category is used to characterize the accuracy of clustering multiple target image patches into any number of candidate categories; select the number of target categories corresponding to the largest clustering index from the multiple candidate categories; and cluster the multiple target image patches into the number of target categories according to the image features of each target image patch to obtain the category of each target image patch.

[0212] In one possible implementation, the merging module 902 is used to determine the position of each target image block in the target image and the category of each target image block; for any two target image blocks, if the two target image blocks are of the same category and are adjacent in the target image, then the two target image blocks are merged into one target image region; if the two target image blocks are of different categories, or are not adjacent in the target image, or are of different categories and are not adjacent in the target image, then the two target image blocks are merged into two target image regions.

[0213] In one possible implementation, the determining module 903 is used to, for any target image region, acquire at least one of the image features, category, and size information of each target image block in the target image region; and determine the image features of the target image region based on at least one of the image features, category, and size information of each target image block in the target image region.

[0214] In one possible implementation, the determining module 903 is used to determine a graph structure based on the image features of each target image region and the position of each target image region in the target image. The graph structure includes at least two nodes and at least one edge. A node represents the image feature of a target image region, and an edge represents the positional adjacency of the target image regions corresponding to the two nodes at both ends of an edge. The category of the target image is determined based on the graph structure.

[0215] In one possible implementation, the device further includes:

[0216] The merging module 902 is also used to merge multiple sample image blocks into at least two sample image regions based on the correlation between multiple sample image blocks, wherein the sample image regions include at least one sample image block.

[0217] The determination module 903 is also used to determine the image features of each sample image region;

[0218] The acquisition module is used to acquire the second target classification network based on the image features of each sample image region;

[0219] The determination module 903 is used to input the image features of each target image region into the second target classification network to obtain the category of the target image.

[0220] In one possible implementation, an acquisition module is used to acquire the labeled categories of sample images; input the image features of each sample image region into a second initial classification network to obtain the predicted categories of the sample images; and adjust the second initial classification network based on the labeled categories and predicted categories of the sample images to obtain a second target classification network.

[0221] After segmenting a target image into multiple target image blocks, the aforementioned device merges these blocks into at least two target image regions based on the correlation between them. Then, it determines the category of the target image based on the image features of each target image region. This achieves the goal of determining the category of the target image by comprehensively considering the correlation between target image blocks, thus improving the accuracy of image classification results. Furthermore, merging multiple target image blocks into at least two target image regions reduces the number of blocks and improves the efficiency of image classification processing.

[0222] It should be understood that the above Figure 9 The provided device, in implementing its functions, is only illustrated by the division of the above-described functional modules. In practical applications, the functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the device and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation processes are detailed in the method embodiments, and will not be repeated here.

[0223] Figure 10 This diagram illustrates a structural block diagram of a terminal device 1000 provided in an exemplary embodiment of this application. The terminal device 1000 includes a processor 1001 and a memory 1002.

[0224] Processor 1001 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1001 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1001 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1001 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the screen. In some embodiments, processor 1001 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0225] The memory 1002 may include one or more computer-readable storage media, which may be non-transitory. The memory 1002 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1002 are used to store at least one computer program, which is executed by the processor 1001 to implement the image classification method provided in the method embodiments of this application.

[0226] In some embodiments, the terminal device 1000 may also optionally include: a peripheral device interface 1003 and at least one peripheral device. The processor 1001, memory 1002, and peripheral device interface 1003 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1003 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of: a radio frequency circuit 1004, a display screen 1005, a camera assembly 1006, an audio circuit 1007, and a power supply 1008.

[0227] Peripheral device interface 1003 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 1001 and memory 1002. In some embodiments, processor 1001, memory 1002 and peripheral device interface 1003 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1001, memory 1002 and peripheral device interface 1003 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0228] The radio frequency (RF) circuit 1004 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 1004 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 1004 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 1004 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 1004 can communicate with other terminals via at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 1004 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0229] Display screen 1005 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 1005 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 1001 for processing. In this case, display screen 1005 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, display screen 1005 may be a single screen, disposed on the front panel of terminal device 1000; in other embodiments, display screen 1005 may be at least two, disposed on different surfaces of terminal device 1000 or in a folded design; in still other embodiments, display screen 1005 may be a flexible display screen, disposed on a curved or folded surface of terminal device 1000. Furthermore, display screen 1005 may also be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. The display screen 1005 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0230] The camera assembly 1006 is used to acquire images or videos. Optionally, the camera assembly 1006 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 1006 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0231] The audio circuit 1007 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 1001 for processing, or input to the radio frequency circuit 1004 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each located at a different part of the terminal device 1000. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 1001 or the radio frequency circuit 1004 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 1007 may also include a headphone jack.

[0232] The power supply 1008 is used to power the various components in the terminal device 1000. The power supply 1008 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When the power supply 1008 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0233] In some embodiments, the terminal device 1000 further includes one or more sensors 1009. The one or more sensors 1009 include, but are not limited to: an acceleration sensor 1011, a gyroscope sensor 1012, a pressure sensor 1013, an optical sensor 1014, and a proximity sensor 1015.

[0234] Accelerometer 1011 can detect the magnitude of acceleration along the three coordinate axes of a coordinate system established by terminal device 1000. For example, accelerometer 1011 can be used to detect the components of gravitational acceleration along the three coordinate axes. Processor 1001 can control display screen 1005 to display the user interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 1011. Accelerometer 1011 can also be used for games or for acquiring user motion data.

[0235] The gyroscope sensor 1012 can detect the orientation and rotation angle of the terminal device 1000. The gyroscope sensor 1012 can work in conjunction with the accelerometer sensor 1011 to collect the user's 3D movements on the terminal device 1000. Based on the data collected by the gyroscope sensor 1012, the processor 1001 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0236] The pressure sensor 1013 can be disposed on the side bezel of the terminal device 1000 and / or on the lower layer of the display screen 1005. When the pressure sensor 1013 is disposed on the side bezel of the terminal device 1000, it can detect the user's grip signal on the terminal device 1000, and the processor 1001 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 1013. When the pressure sensor 1013 is disposed on the lower layer of the display screen 1005, the processor 1001 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 1005. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0237] An optical sensor 1014 is used to collect ambient light intensity. In one embodiment, the processor 1001 can control the display brightness of the display screen 1005 based on the ambient light intensity collected by the optical sensor 1014. Specifically, when the ambient light intensity is high, the display brightness of the display screen 1005 is increased; when the ambient light intensity is low, the display brightness of the display screen 1005 is decreased. In another embodiment, the processor 1001 can also dynamically adjust the shooting parameters of the camera assembly 1006 based on the ambient light intensity collected by the optical sensor 1014.

[0238] The proximity sensor 1015, also known as a distance sensor, is typically installed on the front panel of the terminal device 1000. The proximity sensor 1015 is used to detect the distance between the user and the front of the terminal device 1000. In one embodiment, when the proximity sensor 1015 detects that the distance between the user and the front of the terminal device 1000 is gradually decreasing, the processor 1001 controls the display screen 1005 to switch from a screen-on state to a screen-off state; when the proximity sensor 1015 detects that the distance between the user and the front of the terminal device 1000 is gradually increasing, the processor 1001 controls the display screen 1005 to switch from a screen-off state to a screen-on state.

[0239] Those skilled in the art will understand that Figure 10 The structure shown does not constitute a limitation on the terminal device 1000, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0240] Figure 11This is a schematic diagram of the server structure provided in the embodiments of this application. The server 1100 can vary considerably due to different configurations or performance. It may include one or more processors 1101 and one or more memories 1102, wherein the one or more memories 1102 store at least one computer program, which is loaded and executed by the one or more processors 1101 to implement the image classification method provided in the above-described method embodiments. For example, the processor 1101 is a CPU. Of course, the server 1100 may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server 1100 may also include other components for implementing device functions, which will not be elaborated here.

[0241] In an exemplary embodiment, a computer-readable storage medium is also provided, which stores at least one computer program that is loaded and executed by a processor to enable an electronic device to implement any of the above-described image classification methods.

[0242] Optionally, the aforementioned computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0243] In an exemplary embodiment, a computer program or computer program product is also provided, which stores at least one computer program that is loaded and executed by a processor to enable the computer to implement any of the above-described image classification methods.

[0244] It should be understood that "multiple" as used in this article refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0245] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0246] The above description is merely an exemplary embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. An image classification method, characterized by, The method includes: The target image to be classified is divided into multiple target image patches; Determine the category of each target image patch; For any two target image blocks, if the two target image blocks are of the same category, the two target image blocks are merged into one target image region; if the two target image blocks are of different categories, the two target image blocks are merged into two target image regions, and the target image region includes at least one target image block. Determine the image features of each target image region; The category of the target image is determined based on the image features of each target image region.

2. The method of claim 1, wherein, Determining the category of each target image patch includes: The image features of each target image block are obtained, and the image features of any target image block are used to describe the texture information of the target image block. The category of each target image patch is determined based on its image features.

3. The method of claim 2, wherein, Before acquiring the image features of each target image patch, the method further includes: The sample image is segmented into multiple sample image blocks; A target feature extraction network is obtained based on the multiple sample image blocks; The step of obtaining the image features of each target image patch includes: The target feature extraction network is used to obtain the image features of each target image block.

4. The method of claim 3, wherein, The target feature extraction network obtained from the plurality of sample image patches includes: Perform a first data augmentation process on each sample image block to obtain each sample image block after the first data augmentation process; Each sample image block is subjected to a second data augmentation process to obtain each sample image block after the second data augmentation process. The image features of each sample image block after the first data augmentation process and the image features of each sample image block after the second data augmentation process are obtained based on the initial feature extraction network; Based on the image features of the sample image blocks after the first data augmentation process and the image features of the sample image blocks after the second data augmentation process, the initial feature extraction network is adjusted to obtain the target feature extraction network.

5. The method of claim 2, wherein, The step of obtaining the image features of each target image patch includes: For any target image block, the color histogram or orientation gradient histogram of the target image block is calculated based on the pixel information of each pixel in the target image block to obtain the image features of the target image block.

6. The method according to claim 2, characterized in that, before determining the category of each target image patch based on the image features of each target image patch, it further includes: Obtain image features from multiple sample image patches; The first target classification network is obtained based on the image features of each sample image block; Determining the category of each target image patch based on its image features includes: The image features of each target image patch are input into the first target classification network to obtain the category of each target image patch.

7. The method of claim 6, wherein, The step of obtaining the first target classification network based on the image features of each sample image patch includes: Based on the image features of each sample image block, the multiple sample image blocks are clustered to obtain the first category of each sample image block; The image features of each sample image block are input into a first initial classification network to obtain the second category of each sample image block; The first initial classification network is adjusted based on the first category and the second category of each sample image block to obtain the first target classification network.

8. The method according to claim 2, characterized in that determining the category of each target image patch based on the image features of each target image patch includes: Based on the image features of each target image block, clustering is performed on the multiple target image blocks to obtain the category of each target image block.

9. The method of claim 8, wherein, The step of clustering the multiple target image patches according to their image features to obtain the category of each target image patch includes: A clustering index for the number of multiple candidate categories is obtained, and the clustering index for the number of any candidate category is used to characterize the accuracy of clustering the multiple target image patches into the number of categories of the any candidate category. Select the number of target categories corresponding to the largest clustering index from the plurality of candidate categories; Based on the image features of each target image patch, the multiple target image patches are clustered into the target category number of categories to obtain the category of each target image patch.

10. The method of claim 1, wherein, The method further includes: Determine the position of each target image block in the target image; If any two target image blocks are of the same category and are not adjacent in the target image, then the two target image blocks are merged into two target image regions. If any two target image patches belong to the same category, then merging the two target image patches into one target image region includes: If any two target image blocks are of the same category and are adjacent in position in the target image, then the two target image blocks are merged into one target image region. If any two target image patches belong to different categories, then merging the two target image patches into two target image regions includes: If any two target image blocks are of different categories, or if any two target image blocks are of different categories and their positions in the target image are not adjacent, then the two target image blocks will be merged into two target image regions.

11. The method according to any one of claims 1 to 10, characterized in that, The process of determining the image features of each target image region includes: For any target image region, obtain at least one of the image features, category, and size information of each target image block in the target image region; The image features of any target image region are determined based on at least one of the image features, category, and size information of each target image block in the target image region.

12. The method according to any one of claims 1 to 10, characterized in that, Determining the category of the target image based on the image features of each target image region includes: A graph structure is determined based on the image features of each target image region and the position of each target image region in the target image. The graph structure includes at least two nodes and at least one edge. A node represents the image feature of a target image region, and an edge represents the positional adjacency of the target image regions corresponding to the two nodes at both ends of the edge. The category of the target image is determined based on the graph structure.

13. The method according to any one of claims 1 to 10, characterized in that, Before determining the category of the target image based on the image features of each target image region, the method further includes: Based on the correlation between multiple sample image blocks, the multiple sample image blocks are merged into at least two sample image regions, and the sample image regions include at least one sample image block; Determine the image features of each sample image region; A second target classification network is obtained based on the image features of each sample image region; Determining the category of the target image based on the image features of each target image region includes: The image features of each target image region are input into the second target classification network to obtain the category of the target image.

14. The method of claim 13, wherein, The second target classification network obtained based on the image features of each sample image region includes: Obtain the labeled category of the sample image; The image features of each sample image region are input into the second initial classification network to obtain the predicted category of the sample image; Based on the labeled category and the predicted category of the sample image, the second initial classification network is adjusted to obtain the second target classification network.

15. An image classification apparatus characterized by comprising: The device includes: The segmentation module is used to segment the target image to be classified into multiple target image patches; The merging module is used to merge any two target image blocks into one target image region if the two target image blocks are of the same category; and to merge the two target image blocks into two target image regions if the two target image blocks are of different categories, wherein the target image region includes at least one target image block. The determination module is used to determine the image features of each target image region; The determining module is further configured to determine the category of the target image based on the image features of each target image region.

16. The apparatus of claim 15, wherein, The merging module is used to acquire the image features of each target image block, wherein the image features of any target image block are used to describe the texture information of the target image block; and to determine the category of each target image block based on the image features of each target image block.

17. The apparatus of claim 16, wherein, The device further includes: The segmentation module is also used to segment the sample image into multiple sample image blocks; The acquisition module is used to acquire the target feature extraction network based on the multiple sample image blocks; The merging module is used to obtain the image features of each target image block based on the target feature extraction network.

18. The apparatus of claim 17, wherein, The acquisition module is configured to perform a first data augmentation process on each sample image block to obtain each sample image block after the first data augmentation process; perform a second data augmentation process on each sample image block to obtain each sample image block after the second data augmentation process; acquire image features of each sample image block after the first data augmentation process and image features of each sample image block after the second data augmentation process based on an initial feature extraction network; and adjust the initial feature extraction network based on the image features of each sample image block after the first data augmentation process and image features of each sample image block after the second data augmentation process to obtain the target feature extraction network.

19. The apparatus according to claim 16, characterized in that, The merging module is used to calculate the color histogram or orientation gradient histogram of any target image block based on the pixel information of each pixel in the target image block, thereby obtaining the image features of the target image block.

20. The apparatus according to claim 16, characterized in that, The device further includes: The acquisition module is used to acquire image features from multiple sample image blocks; and to acquire the first target classification network based on the image features of each sample image block. The merging module is used to input the image features of each target image block into the first target classification network to obtain the category of each target image block.

21. The apparatus according to claim 20, characterized in that, The acquisition module is configured to perform clustering processing on the plurality of sample image blocks according to the image features of each sample image block to obtain a first category of each sample image block; input the image features of each sample image block into a first initial classification network to obtain a second category of each sample image block; and adjust the first initial classification network according to the first category and the second category of each sample image block to obtain a first target classification network.

22. The apparatus according to claim 16, characterized in that, The merging module is used to perform clustering processing on the multiple target image blocks according to the image features of each target image block to obtain the category of each target image block.

23. The apparatus according to claim 22, characterized in that, The merging module is used to obtain a clustering index for the number of multiple candidate categories. The clustering index for the number of any candidate category is used to characterize the accuracy of clustering the multiple target image patches into the number of categories of the specified candidate category. Select the number of target categories corresponding to the largest clustering index from the multiple candidate category numbers; based on the image features of each target image patch, cluster the multiple target image patches into the number of target categories to obtain the category of each target image patch.

24. The apparatus according to claim 15, characterized in that, The merging module is further configured to determine the position of each target image block in the target image; if any two target image blocks are of the same category and are not adjacent in the target image, then the two target image blocks are merged into two target image regions; if any two target image blocks are of the same category and are adjacent in the target image, then the two target image blocks are merged into one target image region; if any two target image blocks are of different categories, or if any two target image blocks are of different categories and are not adjacent in the target image, then the two target image blocks are merged into two target image regions.

25. The apparatus according to any one of claims 15 to 24, characterized in that, The determining module is configured to, for any target image region, acquire at least one of the image features, category, and size information of each target image block in the target image region; and determine the image features of the target image region based on at least one of the image features, category, and size information of each target image block in the target image region.

26. The apparatus according to any one of claims 15 to 24, characterized in that, The determining module is used to determine a graph structure based on the image features of each target image region and the position of each target image region in the target image. The graph structure includes at least two nodes and at least one edge. A node represents the image feature of a target image region, and an edge represents the positional adjacency of the target image regions corresponding to the two nodes at both ends of the edge. The module determines the category of the target image based on the graph structure.

27. The apparatus according to any one of claims 15 to 24, characterized in that, The device further includes: The merging module is further configured to merge the multiple sample image blocks into at least two sample image regions based on the correlation between the multiple sample image blocks, wherein the sample image region includes at least one sample image block; The determining module is also used to determine the image features of each sample image region; The acquisition module is used to acquire a second target classification network based on the image features of each sample image region; The determining module is used to input the image features of each target image region into the second target classification network to obtain the category of the target image.

28. The apparatus according to claim 27, characterized in that, The acquisition module is used to acquire the labeled category of the sample image; input the image features of each sample image region into the second initial classification network to obtain the predicted category of the sample image; Based on the labeled category and the predicted category of the sample image, the second initial classification network is adjusted to obtain the second target classification network.

29. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing at least one computer program, which is loaded and executed by the processor to enable the electronic device to implement the image classification method as described in any one of claims 1 to 14.

30. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to enable the computer to implement the image classification method as described in any one of claims 1 to 14.

31. A computer program product, characterized in that, The computer program product stores at least one computer program, which is loaded and executed by a processor to enable the computer to implement the image classification method as described in any one of claims 1 to 14.

Images