An image processing method and related apparatus

Abstract

The application discloses an image processing method applied to the field of artificial intelligence, and comprises the following steps: processing a first image through a first network to obtain a plurality of feature maps with different resolutions; searching in a first feature map with low resolution through a second network based on a second image to obtain a first region in the first feature map, the first region being used for changing a classification result of the first feature map; obtaining a mapping region of the first region in a second feature map; searching in the mapping region through the second network based on the second image to obtain a second region in the second feature map with high resolution; and obtaining a region corresponding to the second region in the first image according to the second region. In the scheme, the region is searched on the low-resolution feature map first, and then the region is searched in the corresponding region on the high-resolution feature map, so that the region needing to be searched on the high-resolution feature map is effectively reduced, the efficiency is improved, and the time of output result is reduced.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an image processing method and related apparatus. Background Technology

[0002] Artificial intelligence (AI) is the theory, methods, technology, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new kind of intelligent machine that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess the functions of perception, reasoning, and decision-making.

[0003] In Explainable Artificial Intelligence (XAI), counterfactual explanation is a method of explanation that aligns relatively well with human cognition. A counterfactual explanation refers to the minimum condition that needs to be changed to make an event produce a different outcome. Based on counterfactual explanation, we can understand how an instance can be changed to alter its predicted result.

[0004] Applying counterfactual interpretation to classification problems in computer vision involves perturbing the original image to generate a counterfactual image or feature map, thereby altering the category of the generated image. That is, if the input image is the generated counterfactual image or feature map, then the classification result of the input image will be the category expected by the user, which differs from the category of the original image.

[0005] Currently, the common method for obtaining counterfactual explanations is to replace regions in the input image one by one with regions in the comparison image, and then infer which region replacement will change the classification result of the synthesized image, ultimately obtaining the synthesized image that changes the classification result. However, current methods for obtaining counterfactual explanations require region-by-region replacement on high-resolution images, which is inefficient and often takes a long time to output results. Summary of the Invention

[0006] This application provides an image processing method that acquires multiple feature maps of different resolutions from an input image, searches for key regions on the lower-resolution feature map to change the classification result, determines the mapping region of the key region on the lower-resolution feature map to the higher-resolution feature map, and further searches for key regions on this mapping region, ultimately obtaining the region on the image that needs to be replaced. By first searching for key regions on the low-resolution feature map and then searching for the corresponding region on the high-resolution feature map, the search area on the high-resolution feature map is effectively narrowed down, improving efficiency and reducing the time required to output the result.

[0007] This application provides an image processing method, comprising: firstly, processing a first image through a first network to obtain multiple feature maps with different resolutions, wherein the multiple feature maps include a first feature map and a second feature map, the resolution of the second feature map being greater than the resolution of the first feature map. The first network may include multiple feature extraction layers connected sequentially. After the first image is input into the first network, the multiple feature extraction layers perform feature processing on the first image, outputting feature maps of different resolutions respectively. For example, for the first feature extraction layer in the first network, the input of the first feature extraction layer is the first image; after extracting features from the first image, the first feature extraction layer outputs a feature map. For any feature extraction layer in the first network other than the first feature extraction layer, the input of the feature extraction layer is the feature map output by the previous feature extraction layer; after processing the input feature map, the feature extraction layer outputs a processed feature map, which is then used as the input of the next feature extraction layer. This process continues, with the multiple feature extraction layers in the first network outputting feature maps of different resolutions.

[0008] Then, based on the second image, a second network searches within the first feature map to obtain a first region in the first feature map. Specifically, the second network may include a classification network capable of predicting the category of the feature map based on it, thus obtaining a classification result. The classification results of the first image and the second image are different. The first region is used to change the classification result of the first feature map. After replacing the first region in the first feature map with the corresponding region in the second image, a third image (i.e., the first feature map after replacing the first region) is obtained. The classification result of the third image is the same as that of the second image. The first region can be understood as a key region in the first feature map. A key region in a feature map refers to a region that can affect the classification result of the feature map. When a key region in a feature map is replaced with a region in a specific image, the classification result of the feature map after replacing the key region is the same as the classification result of that specific image.

[0009] Searching in the first feature map means replacing each region in the first feature map with a corresponding region in the second image, in order to determine which region in the first feature map, after being replaced, will increase the probability that the classification network will predict the category to which the second image belongs.

[0010] Next, the mapping region of the first region in the second feature map is obtained, and the content of the first region is related to the mapping region. Simply put, the second feature extraction layer in the first network outputs the second feature map, and this second feature map is input into the feature extraction layer that outputs the first feature map. After processing the second feature map, the feature extraction layer that outputs the first feature map outputs the first feature map. The first region in the first feature map is obtained by processing the mapping region in the second feature map by the feature extraction layer that outputs the first feature map.

[0011] Based on the second image, the second network searches the mapped region to obtain a second region in the second feature map. The second region may include one or more regions. After replacing the second region in the second feature map with the corresponding region in the second image, a second feature map with the replaced second region is obtained. The classification result of this second feature map with the replaced second region is the same as the classification result of the second image. That is, the second region is the key region in the second feature map.

[0012] Finally, based on the second region, the region in the first image corresponding to the second region is obtained, so as to output the first image marked with the corresponding region.

[0013] In this embodiment, multiple feature maps of different resolutions of the input image are acquired. Key regions for altering the classification results of the lower-resolution feature maps are searched for. Then, the mapping regions of these key regions on the lower-resolution feature maps are determined, and further searches are conducted on these mapping regions to obtain the areas in the image that need to be replaced. By first searching for key regions on the low-resolution feature maps and then searching for corresponding regions on the high-resolution feature maps, the search area on the high-resolution feature maps is effectively narrowed down, improving efficiency and reducing output time.

[0014] In one possible implementation, the first feature map is the feature map with the lowest resolution among the plurality of feature maps. Among the plurality of feature maps, the first feature map has the fewest regions to be searched. Therefore, compared to the other feature maps among the plurality of feature maps, the regions in the first feature map can be searched in the least amount of time.

[0015] In one possible implementation, the resolution of the feature maps output by the multiple feature extraction layers in the first network gradually decreases from the first feature extraction layer to the last feature extraction layer. The second feature map is the feature map among the multiple feature maps whose resolution is closest to that of the first feature map. That is, the feature extraction layer used to output the second feature map and the feature extraction layer used to output the first feature map can be two adjacent feature extraction layers. For example, if the first feature map is the feature map with the lowest resolution among the multiple feature maps, then the feature extraction layer used to output the first feature map is the last feature extraction layer of the first network, and the feature extraction layer used to output the second feature map is the second-to-last feature extraction layer in the first network.

[0016] In one possible implementation, the resolution of the second feature map is a first preset resolution, meaning the second feature map is extracted by a feature extraction layer capable of extracting a feature map at the first preset resolution. Optionally, the resolution of the second feature map may also be within a preset resolution range. Alternatively, the second feature map may be a feature map output by a target feature extraction layer, which is the feature extraction layer among the multiple feature extraction layers of the first network whose output resolution is closest to the first preset resolution.

[0017] In practical applications, users can adjust the first preset resolution according to their actual needs. For example, when a user wants to obtain more accurate output results, the user can adjust the first preset resolution to a larger resolution; when a user wants to obtain output results faster, the user can adjust the first preset resolution to a smaller resolution.

[0018] In one possible implementation, the second feature map and the first feature map are further separated by one or more feature maps from the plurality of feature maps, wherein the resolution of the one or more feature maps is greater than the resolution of the first feature map and less than the resolution of the second feature map. That is, in the first network, the feature extraction layer used to output the second feature map and the feature extraction layer used to output the first feature map are not adjacent feature extraction layers, and one or more feature extraction layers are separated from the feature extraction layer used to output the second feature map and the feature extraction layer used to output the first feature map.

[0019] In one possible implementation, obtaining the mapping region of the first region in the second feature map includes: obtaining the kernel size and stride of the feature extraction layer used to extract the first feature map; and determining the mapping region in the second feature map based on the first region and the kernel size and stride of the feature extraction layer. That is, the mapping region is actually the receptive field of the first region in the second feature map. This receptive field can be obtained based on the kernel size and stride of the feature extraction layer used to extract the first feature map and the first region.

[0020] In one possible implementation, to maximize the accuracy of the output, region search can be performed map-by-map among the obtained feature maps, with the search range of each feature map being a mapped region of the region already searched in the previous feature map. The first feature map and the second feature map are two of the feature maps used for region search.

[0021] Specifically, the method further includes: repeatedly executing the region search step until a termination condition is met to obtain a target region; and outputting the region in the first image corresponding to the target region. The region search step includes: obtaining the mapping region of the region in the i-th feature map to the (i+1)-th feature map, wherein the region in the i-th feature map is used to change the classification result of the i-th feature map; based on the second image, searching the mapping region in the (i+1)-th feature map through the second network to obtain the region in the (i+1)-th feature map, wherein the region in the (i+1)-th feature map is used to change the classification result of the (i+1)-th feature map; and incrementing the value of i by 1. Wherein, the target region is the region in the (i+1)-th feature map when the termination condition is met, and the resolution of the i-th feature map is less than the resolution of the (i+1)-th feature map. i is greater than or equal to 2, and when i equals 2, the i-th feature map is the second feature map.

[0022] In simple terms, after a key region is found in the feature map each time, the corresponding mapping region is determined in the feature map extracted by the previous feature extraction layer, and the search for key regions continues within the mapping region until the termination condition is met.

[0023] In one possible implementation, the termination condition includes the resolution of the (i+1)th feature map reaching a second preset resolution. It should be understood that a higher resolution feature map used for the search region contains less information per unit area, resulting in a more accurate search region. However, this also requires a larger search area and more time. Therefore, in practical applications, a balance can be struck between the accuracy of the results and computational efficiency based on the user's needs to determine the final resolution of the searched feature map.

[0024] In one possible implementation, the termination condition includes the area ratio between the region in the (i+1)th feature map and the mapped region in the (i+1)th feature map being greater than a preset threshold. It should be understood that during the region search process, after searching the region in the i-th feature map, continuing to search for a region in the mapped region of the (i+1)th feature map is actually to further narrow down the region's range to obtain a more accurate result. When the search finds that the area ratio between the region in the (i+1)th feature map and the mapped region in the (i+1)th feature map is greater than the preset threshold, it can be considered that the region cannot be narrowed down by continuing to search for a region in the mapped region of the (i+1)th feature map, and therefore the region search can be terminated.

[0025] In one possible implementation, the step of searching the first feature map using a second network based on the second image to obtain a first region in the first feature map includes: processing the second image using the first network to obtain a third feature map, the resolution of which is the same as that of the first feature map; replacing a portion of the first feature map with a portion of the third feature map, such that the classification result of the first feature map after the partial region replacement is the same as the classification result of the second image; and determining the replaced region in the first feature map after the partial region replacement as the first region.

[0026] In other words, the region corresponding to the second image mentioned above can refer to a part of the third feature map corresponding to the second image, and the resolution of the third feature map corresponding to the second image is the same as the resolution of the first feature map.

[0027] In one possible implementation, the step of searching for a region in the first feature map based on the second image using a second network to obtain a first region in the first feature map includes: obtaining a non-background region in the first feature map, wherein the non-background region is the region in the first feature map other than the background region; and searching in the non-background region of the first feature map using the second network based on the second image. In this application, the non-background region in the image can refer to the region where the target to be classified is located, and the background region in the image can refer to the region other than the region where the target to be classified is located. For example, assuming the target to be classified in the first feature map is a bird, then the non-background region in the first feature map is the region where the bird is located. The background region in the first feature map is the region outside the region where the bird is located, such as the region where scenery like the sky, grass, or forest is located. As another example, assuming the target to be classified in the first feature map is a car, then the non-background region in the first feature map is the region where the car is located. The background region in the first feature map is the region outside the region where the car is located, such as the region where objects like roads or courtyards are located.

[0028] In this scheme, before performing the search of the region, the background region in the input image is identified by an algorithm, thereby excluding the background region in the input image from the search range and reducing the search range of the region.

[0029] In one possible implementation, the search in the first feature map based on the second image via the second network includes: obtaining a marked region in the first image, which may be a region marked by the user to indicate a region where a key region search is not required; and searching in a third region in the first feature map based on the second image via the second network, where the third region is a region in the first feature map other than a fourth region, and the fourth region is a region in the first feature map corresponding to the marked region.

[0030] In one possible implementation, obtaining the region in the first image corresponding to the second region based on the second region includes: obtaining the resolution of the second feature map and the resolution of the first image; and determining the region in the first image corresponding to the second region based on the position of the second region in the second feature map, the resolution of the second feature map, and the resolution of the first image.

[0031] In one possible implementation, the second region includes multiple sub-regions. The step of obtaining the region in the first image corresponding to the second region based on the second region includes: obtaining the probability corresponding to each of the multiple sub-regions, where the probability is the increase in probability when the sub-region is predicted as the category of the second image after the second feature map replaces it; sorting the probabilities corresponding to each sub-region in descending order to obtain a sorting result; and obtaining multiple regions in the first image corresponding to the multiple sub-regions based on the sorting result and the multiple sub-regions, wherein the multiple regions have different labeling methods, and the labeling methods are related to the sorting result.

[0032] In this scheme, regions are sorted and displayed according to their contribution to changing the classification results of the second feature map, so as to highlight the importance of different regions.

[0033] A second aspect of this application provides an image processing apparatus, including an acquisition unit and a processing unit. The processing unit is configured to process a first image through a first network to obtain multiple feature maps, the multiple feature maps including a first feature map and a second feature map, wherein the resolution of the second feature map is greater than the resolution of the first feature map; the processing unit is further configured to search within the first feature map based on a second image through a second network to obtain a first region in the first feature map, wherein the first region is used to change the classification result of the first feature map, the classification result of the first image is different from that of the second image, and the classification result of a third image is the same as that of the second image, the third image being a first feature map after replacing the first region with a corresponding region in the second image; the acquisition unit is configured to acquire a mapping region of the first region in the second feature map, the content of the first region being related to the mapping region; the processing unit is further configured to search within the mapping region based on the second image through the second network to obtain a second region in the second feature map, the second region being used to change the classification result of the first feature map; the processing unit is further configured to obtain a region in the first image corresponding to the second region based on the second region.

[0034] In one possible implementation, the first feature map is the feature map with the smallest resolution among the plurality of feature maps.

[0035] In one possible implementation, the second feature map is the feature map among the plurality of feature maps whose resolution is closest to that of the first feature map; or, the resolution of the second feature map is a first preset resolution; or, the second feature map and the first feature map are further separated by one or more feature maps among the plurality of feature maps, the resolution of the one or more feature maps being greater than the resolution of the first feature map and less than the resolution of the second feature map.

[0036] In one possible implementation, the acquisition unit is configured to: acquire the kernel size and stride of the feature extraction layer used to output the first feature map; and determine the mapping region in the second feature map based on the first region and the kernel size and stride of the feature extraction layer.

[0037] In one possible implementation, the processing unit is further configured to: repeatedly execute the region search step until a termination condition is met to obtain a target region; output the region in the first image corresponding to the target region; the region search step includes: obtaining the mapping region of the region in the i-th feature map in the (i+1)-th feature map, the region in the i-th feature map being used to change the classification result of the i-th feature map; based on the second image, performing a region search in the mapping region of the (i+1)-th feature map through the second network to obtain the region in the (i+1)-th feature map, the region in the (i+1)-th feature map being used to change the classification result of the (i+1)-th feature map; incrementing the value of i by 1; wherein, the target region is the region in the (i+1)-th feature map when the termination condition is met, the resolution of the i-th feature map is less than the resolution of the (i+1)-th feature map, i is greater than or equal to 2, and when i is equal to 2, the i-th feature map is the second feature map.

[0038] In one possible implementation, the termination condition includes the resolution of the (i+1)th feature map reaching a second preset resolution.

[0039] In one possible implementation, the termination condition includes a ratio of the area between the region in the (i+1)th feature map and the mapped region in the (i+1)th feature map being greater than a preset threshold.

[0040] In one possible implementation, the processing unit is further configured to: process the second image through the first network to obtain a third feature map, the resolution of the third feature map being the same as the resolution of the first feature map; replace a portion of the first feature map with a portion of the third feature map, such that the classification result of the first feature map after replacing the portion of the region is the same as the classification result of the second image; and determine the replaced region in the first feature map after replacing the portion of the region as the first region.

[0041] In one possible implementation, the acquisition unit is further configured to acquire a non-background region in the first feature map, wherein the non-background region is a region in the first feature map other than the background region; the processing unit is further configured to search for the non-background region in the first feature map based on the second image using the second network.

[0042] In one possible implementation, the acquisition unit is further configured to acquire a marked region in the first image; the processing unit is further configured to search for a third region in the first feature map based on the second image using the second network, wherein the third region is a region in the first feature map other than the fourth region, and the fourth region is a region in the first feature map corresponding to the marked region.

[0043] In one possible implementation, the processing unit is further configured to determine the region in the first image corresponding to the second region based on the position of the second region in the second feature map, the resolution of the second feature map, and the resolution of the first image.

[0044] In one possible implementation, the second region includes multiple sub-regions; the processing unit is further configured to: obtain the probability corresponding to each sub-region in the multiple sub-regions, the probability being the increased probability when the sub-region is predicted as the category of the second image after the second feature map replaces it; sort the probabilities corresponding to each sub-region in descending order to obtain a sorting result; and obtain multiple regions in the first image corresponding to the multiple sub-regions based on the sorting result and the multiple sub-regions, the multiple regions having different labeling methods, the labeling methods being related to the sorting result.

[0045] The image processing apparatus provided in the second aspect corresponds to the method described in the first aspect and is used to implement or cooperate with the implementation of the method provided in the first aspect. Therefore, it can achieve the same or corresponding beneficial effects as the first aspect, which will not be elaborated here.

[0046] A third aspect of this application provides an image processing apparatus, which may include a processor coupled to a memory. The memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the method described in the first aspect is implemented. For details regarding the steps of the various possible implementations of the first aspect executed by the processor, please refer to the first aspect; further details will not be repeated here.

[0047] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the method described in the first aspect.

[0048] A fifth aspect of this application provides a circuit system including a processing circuit configured to perform the method described in the first aspect above.

[0049] The sixth aspect of this application provides a computer program product that, when run on a computer, causes the computer to perform the method described in the first aspect above.

[0050] A seventh aspect of this application provides a chip system including a processor for supporting a server or threshold acquisition device in implementing the functions involved in the first aspect above, such as transmitting or processing data and / or information involved in the methods described above. In one possible design, the chip system further includes a memory for storing program instructions and data necessary for the server or communication device. This chip system may be composed of chips or may include chips and other discrete devices. Attached Figure Description

[0051] Figure 1 A structural diagram illustrating the main framework of artificial intelligence;

[0052] Figure 2 A schematic diagram of a convolutional neural network provided in an embodiment of this application;

[0053] Figure 3 A schematic diagram of a convolutional neural network provided in an embodiment of this application;

[0054] Figure 4 A schematic diagram of a system architecture provided for an embodiment of this application;

[0055] Figure 5 This is a schematic diagram illustrating an image replacement operation according to an embodiment of this application;

[0056] Figure 6(a) is a schematic diagram of an application scenario in an educational product provided by an embodiment of this application;

[0057] Figure 6(b) is a schematic diagram of an application scenario in AI program development provided by an embodiment of this application;

[0058] Figure 6(c) is a schematic diagram of another application scenario in AI program development provided by the embodiments of this application;

[0059] Figure 7 A schematic flowchart of an image processing method provided in an embodiment of this application;

[0060] Figure 8(a) is a schematic diagram of a calculation of the receptive field provided in an embodiment of this application;

[0061] Figure 8(b) is a schematic diagram of converting a second feature map into a first image according to an embodiment of this application;

[0062] Figure 9 A schematic diagram of a search area provided in an embodiment of this application;

[0063] Figure 10(a) is a schematic diagram of performing a region search based on a locked region according to an embodiment of this application;

[0064] Figure 10(b) is a comparative schematic diagram of performing a region search based on a locked region according to an embodiment of this application;

[0065] Figure 11 A schematic flowchart of an image processing method provided in an embodiment of this application;

[0066] Figure 12 A schematic diagram illustrating an example of a key search region provided in this application embodiment;

[0067] Figure 13 A schematic diagram illustrating a layer-by-layer search for key regions provided in this application embodiment;

[0068] Figure 14 A comparative schematic diagram of a key area provided for an embodiment of this application;

[0069] Figure 15 An image processing apparatus provided in an embodiment of this application;

[0070] Figure 16 A schematic diagram of the structure of the execution device provided in the embodiments of this application;

[0071] Figure 17 This is a schematic diagram of a chip structure provided in an embodiment of this application. Detailed Implementation

[0072] The embodiments of the present invention will now be described with reference to the accompanying drawings. The terminology used in the embodiments section is for illustrative purposes only and is not intended to limit the scope of the invention.

[0073] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.

[0074] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0075] For ease of understanding, the technical terms involved in the embodiments of this application will be explained below.

[0076] Receptive Field: In convolutional neural networks (CNNs), the receptive field refers to the size of the region on the input image / input feature map that corresponds to the pixels in the feature map output by each layer of the CNN. In other words, the points on the feature map are calculated from all pixels within the receptive field of the input image. A larger receptive field means that the points on the feature map can access a wider range of the original image, resulting in more global and semantically higher-level features. Conversely, a smaller receptive field indicates that the features contained in the points on the feature map are more localized and detailed.

[0077] First, the overall workflow of the artificial intelligence system is described; please refer to [link / reference]. Figure 1 , Figure 1 The diagram illustrates a structural framework for artificial intelligence (AI). The framework is further elaborated below along two dimensions: the "Intelligent Information Chain" (horizontal axis) and the "IT Value Chain" (vertical axis). The "Intelligent Information Chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT Value Chain" reflects the value that AI brings to the information technology industry, from the underlying infrastructure of human intelligence and information (provided and processed through technological means) to the industrial ecosystem of the system.

[0078] (1) Infrastructure.

[0079] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. This communication occurs through sensors; computing power is provided by intelligent chips (hardware acceleration chips such as CPUs, NPUs, GPUs, ASICs, and FPGAs); and the basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0080] (2) Data.

[0081] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0082] (3) Data processing.

[0083] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0084] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0085] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0086] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0087] (4) General ability.

[0088] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0089] (5) Smart products and industry applications.

[0090] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent terminals, intelligent transportation, intelligent healthcare, autonomous driving, smart cities, etc.

[0091] The method provided in this application is described below from the perspectives of model training and model application:

[0092] The model training method provided in this application can be applied to data processing methods such as data training, machine learning, and deep learning. It performs symbolic and formal intelligent information modeling, extraction, preprocessing, and training on training data to finally obtain a trained neural network model (such as the target neural network model in this application embodiment). The target neural network model can be used for model inference. Specifically, input data can be input into the target neural network model to obtain output data.

[0093] Since the embodiments of this application involve a large number of neural network applications, for ease of understanding, the relevant terms and concepts such as neural networks involved in the embodiments of this application will be introduced below.

[0094] (1) Neural network.

[0095] A neural network can be composed of neural units, which can be defined as a computational unit that takes xs (i.e., input data) and an intercept of 1 as input. The output of this computational unit can be:

[0096] Where s = 1, 2, ..., n, where n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer, and the activation function can be the sigmoid function. A neural network is a network formed by connecting multiple of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0097] (2) A Convolutional Neural Network (CNN) is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers. This feature extractor can be viewed as a filter, and the convolution process can be seen as using a trainable filter to convolve with an input image or a convolutional feature map. A convolutional layer refers to the layer of neurons in a CNN that performs convolution processing on the input signal (e.g., the first and second convolutional layers in this embodiment). In a convolutional layer of a CNN, a neuron can be connected to only some of the neurons in neighboring layers. A convolutional layer typically contains several feature planes, each of which can be composed of several rectangularly arranged neural units. Neural units on the same feature plane share weights, which are the convolutional kernels. Shared weights can be understood as the way image information is extracted regardless of location. The underlying principle is that the statistical information of a certain part of the image is the same as that of other parts. This means that image information learned in one part can also be used in another part. Therefore, we can use the same learned image information for all locations in the image. Within the same convolutional layer, multiple convolutional kernels can be used to extract different image information. Generally, the more convolutional kernels there are, the richer the image information reflected by the convolution operation.

[0098] Convolutional kernels can be initialized as matrices of random size, and during the training of a convolutional neural network, they can learn appropriate weights. Furthermore, sharing weights directly reduces the number of connections between layers in the convolutional neural network, while also lowering the risk of overfitting.

[0099] Specifically, such as Figure 2 As shown, the convolutional neural network (CNN) 200 may include an input layer 210, a convolutional / pooling layer 220, wherein the pooling layer is optional, and a neural network layer 230.

[0100] The structure consisting of the convolutional layer / pooling layer 220 and the neural network layer 230 can be the first convolutional layer and the second convolutional layer described in this application. The input layer 210 is connected to the convolutional layer / pooling layer 220, and the convolutional layer / pooling layer 220 is connected to the neural network layer 230. The output of the neural network layer 230 can be input to the activation layer, and the activation layer can perform non-linear processing on the output of the neural network layer 230.

[0101] Convolutional / pooling layers 220. Convolutional layers: such as... Figure 2The convolutional / pooling layer 220 shown may include layers 221-226 as in Examples. In one implementation, layer 221 is a convolutional layer, layer 222 is a pooling layer, layer 223 is a convolutional layer, layer 224 is a pooling layer, layer 225 is a convolutional layer, and layer 226 is a pooling layer. In another implementation, layers 221 and 222 are convolutional layers, layer 223 is a pooling layer, layers 224 and 225 are convolutional layers, and layer 226 is a pooling layer. That is, the output of the convolutional layer can be used as the input of a subsequent pooling layer, or as the input of another convolutional layer to continue the convolution operation.

[0102] Taking convolutional layer 221 as an example, it can include multiple convolution operators, also known as kernels. In image processing, a convolution operator acts as a filter, extracting specific information from the input image matrix. Essentially, a convolution operator can be a weight matrix, which is usually predefined. During the convolution operation, the weight matrix typically processes the input image pixel by pixel (or two pixels by two pixels, depending on the stride) along the horizontal direction, thus extracting specific features from the image. The size of the weight matrix should be related to the image size. It's important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image; during convolution, the weight matrix extends to the entire depth of the input image. Therefore, convolution with a single weight matrix produces a single-depth convolutional output. However, in most cases, multiple weight matrices of the same dimension are applied instead of a single weight matrix. The outputs of each weight matrix are stacked to form the depth dimension of the convolutional image. Different weight matrices can be used to extract different features from an image. For example, one weight matrix can be used to extract image edge information, another weight matrix can be used to extract specific colors from the image, and yet another weight matrix can be used to blur unwanted noise in the image. These multiple weight matrices have the same dimension, and the feature maps extracted by these multiple weight matrices with the same dimension also have the same dimension. The extracted feature maps with the same dimension are then merged to form the output of the convolution operation.

[0103] The weight values ​​in these weight matrices need to be obtained through extensive training in practical applications. The weight matrices formed by the weight values ​​obtained through training can extract information from the input image, thereby helping the convolutional neural network 100 to make correct predictions.

[0104] When a convolutional neural network 200 has multiple convolutional layers, the initial convolutional layers (e.g., 221) tend to extract more general features, which can also be called low-level features. As the depth of the convolutional neural network 200 increases, the features extracted by later convolutional layers (e.g., 226) become more and more complex, such as high-level semantic features. Features with higher semantic levels are more suitable for the problem to be solved.

[0105] Pooling layers: Because it is often necessary to reduce the number of training parameters, pooling layers are often introduced periodically after convolutional layers, such as... Figure 2 In the example of 220, each layer 221-226 can be a convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers.

[0106] Neural Network Layer 230: After processing by the convolutional / pooling layers 220, the convolutional neural network 200 is still insufficient to output the required information. As mentioned earlier, the convolutional / pooling layers 220 only extract features and reduce the parameters introduced by the input image. However, to generate the final output information (the required class information or other relevant information), the convolutional neural network 200 needs to utilize neural network layer 230 to generate one or a set of required class numbers of output. Therefore, neural network layer 230 may include multiple hidden layers (such as...). Figure 2 As shown in layers 231, 232 to 23n) and output layer 240, the parameters contained in these multi-layer hidden layers can be pre-trained based on relevant training data for specific task types, such as image recognition, image classification, image super-resolution reconstruction, etc.

[0107] After the multiple hidden layers in neural network layer 230, the final layer of the entire convolutional neural network 200 is the output layer 240. This output layer 240 has a loss function similar to classification cross-entropy, specifically used to calculate the prediction error. Once the entire convolutional neural network 200 has undergone forward propagation (e.g., ...), the loss function is applied. Figure 2 The propagation from 210 to 240 is completed (forward propagation), and the reverse propagation (such as...) Figure 2 The propagation from 240 to 210 (backpropagation) will begin to update the weight values ​​and biases of the layers mentioned above, in order to reduce the loss of the convolutional neural network 200 and the error between the output of the convolutional neural network 200 through the output layer and the ideal result.

[0108] It should be noted that, as Figure 2 The convolutional neural network 200 shown is merely an example of a convolutional neural network. In specific applications, convolutional neural networks can also exist in the form of other network models, such as... Figure 3The multiple convolutional / pooling layers shown are parallelized, and the extracted features are all input into the full neural network layer 230 for processing.

[0109] (3) Deep neural networks.

[0110] Deep Neural Networks (DNNs), also known as multilayer neural networks, can be understood as neural networks with many hidden layers, though there's no specific metric for "many." DNNs can be categorized into three layers based on their position: input layers, hidden layers, and output layers. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. All layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer. Although DNNs appear complex, the operation of each layer is actually quite simple, resembling a linear relationship as follows: in, It is the input vector. It is the output vector. α is the offset vector, W is the weight matrix (also called coefficients), and α() is the activation function. Each layer is simply an adjustment of the input vector. The output vector is obtained through such a simple operation. Because DNNs have many layers, the coefficients W and the offset vector... The number of these parameters is therefore quite large. The definitions of these parameters in a DNN are as follows: Taking the coefficient W as an example: Assuming a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as... The superscript 3 represents the layer number where coefficient W resides, while the subscript corresponds to the output third layer index 2 and the input second layer index 4. In summary, the coefficients from the k-th neuron in layer L-1 to the j-th neuron in layer L are defined as follows: It's important to note that the input layer does not have a W parameter. In deep neural networks, more hidden layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can perform more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrix of all layers in the trained deep neural network (a weight matrix formed by the vectors W from many layers).

[0111] (4) Loss function.

[0112] In training a deep neural network, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value. Based on the difference, we update the weight vector of each layer (usually pre-configuring parameters before the initial update). For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the deep neural network predicts the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the deep neural network becomes a process of minimizing this loss.

[0113] (5) Backpropagation algorithm.

[0114] Convolutional neural networks can employ backpropagation (BP) to correct the parameters in the initial super-resolution model during training, thereby reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates an error loss; this error loss information is then propagated back to update the parameters in the initial super-resolution model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the super-resolution model, such as the weight matrix.

[0115] (6) Linear operations.

[0116] Linearity refers to a proportional, linear relationship between quantities. Mathematically, it can be understood as a function whose first derivative is a constant. Linear operations include, but are not limited to, addition, null operations, identity operations, convolution, batch normalization (BN), and pooling. Linear operations can also be called linear mappings. A linear mapping must satisfy two conditions: homogeneity and additivity. If either condition is not met, it is nonlinear.

[0117] Homogeneity refers to f(ax) = af(x); additivity refers to the fact that f(x) = ax is linear. It's important to note that x, a, and f(x) are not necessarily scalars; they can be vectors or matrices, forming a linear space of arbitrary dimensions. If x and f(x) are n-dimensional vectors, homogeneity is equivalent to a constant when a is a constant, and additivity is equivalent to a matrix when a is a matrix. Conversely, functions whose graphs are straight lines do not necessarily conform to linear mappings. For example, f(x) = ax + b does not satisfy either homogeneity or additivity, and therefore belongs to nonlinear mappings.

[0118] In the embodiments of this application, the composition of multiple linear operations can be referred to as a linear operation, and each linear operation included in a linear operation can also be referred to as a sub-linear operation.

[0119] Figure 4 This is a schematic diagram of a system architecture provided in an embodiment of this application. Figure 4 In the process, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with external devices. Users can input data to the I / O interface 112 through the client device 140.

[0120] During the preprocessing of input data by the execution device 120, or during the calculation module 111 of the execution device 120 performing calculations and other related processing (such as implementing the neural network function in this application), the execution device 120 may call data, code, etc. in the data storage system 150 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 150.

[0121] Finally, I / O interface 112 returns the processing result to client device 140, thereby providing it to the user.

[0122] Optionally, the customer device 140 may be, for example, a control unit in an autonomous driving system or a functional algorithm module in a mobile terminal, such as a functional algorithm module that can be used to perform related tasks.

[0123] It is worth noting that the training device 120 can generate corresponding target models / rules (such as the target neural network model in this embodiment) based on different training data for different objectives or tasks. The corresponding target models / rules can be used to achieve the above objectives or complete the above tasks, thereby providing the user with the required results.

[0124] exist Figure 4In the scenario shown, the user can manually provide input data, which can be done through the interface provided by I / O interface 112. Alternatively, the client device 140 can automatically send input data to I / O interface 112. If user authorization is required for the client device 140 to automatically send input data, the user can set the corresponding permissions in the client device 140. The user can view the output results of the execution device 110 on the client device 140, which can take the form of display, sound, or action. The client device 140 can also act as a data acquisition terminal, collecting the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130. Alternatively, data can be collected directly from the I / O interface 112 without going through the client device 140, using the input data and output results of the input I / O interface 112 as new sample data and storing them in the database 130.

[0125] It is worth noting that, Figure 4 This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 4 In this context, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed within the execution device 110.

[0126] In Explainable Artificial Intelligence (XAI), counterfactual explanation is a method that aligns relatively well with human cognition. A counterfactual explanation refers to the minimum condition that needs to be changed to cause an event to produce a different outcome. Based on counterfactual explanation, we can understand how an instance can be changed to alter its predicted result. In this way, users can learn how to make predictions align with their expectations by understanding the changes that lead to altered predictions.

[0127] For example, Xiaoming applied for a loan but was rejected by a bank's machine learning-based software. He wants to know why his application was rejected and how he can improve his chances of getting the loan. Based on counterfactual interpretation, the above "why" question can be phrased as: What is the minimum change to a feature (e.g., income, number of credit cards, age, etc.) that would cause the prediction to change from rejection to approval?

[0128] In some possible examples, the counterfactual interpretation is as follows:

[0129] If Xiaoming can earn an extra 10,000 yuan per year, he will be able to get a loan.

[0130] If Xiaoming has few credit cards and has not defaulted on any loans 5 years ago, then he will be able to get a loan.

[0131] In this way, by understanding the conditions that can change the outcome of a loan application, Xiaoming can learn how to make the loan application result closer to his expectations.

[0132] Applying counterfactual interpretation to classification problems in computer vision involves perturbing the original image to generate a counterfactual image or feature map, thereby altering the category of the generated image. That is, if the input image is the generated image, then the classification result of that input image will be the category expected by the user, which differs from the category of the original image.

[0133] For example, given an image of interest belonging to category A, a human might want to know why this image of interest doesn't belong to category B. Therefore, a set of experimental images can be input, including the image of interest (belonging to category A) and distractor images (belonging to category B). By replacing certain regions on the image of interest with corresponding locations on the distractor images, a synthetic image is obtained. Furthermore, the classification model's classification result for this synthetic image changes from category A to category B.

[0134] In simple terms, the application of counterfactual interpretation in image classification is as follows: obtain an input image and a comparison image, replace each region in the input image with a region in the comparison image, and infer which region replacement will change the classification result of the synthesized image, ultimately obtaining a synthesized image that changes the classification result.

[0135] For example, see Figure 5 , Figure 5 This is a schematic diagram illustrating an image replacement operation according to an embodiment of this application. Figure 5As shown, for input image I and comparison image I', the category of input image I is Crested Auk, and the category of comparison image I' is Red-faced Cormorant. During processing, all regions in input image I are used as search candidate regions. For any given search candidate region, all regions in comparison image I' need to be replaced one by one, and the classification result of the synthesized image after the replacement regions is inferred. For example, assuming the resolution of input image I is 100×100, input image I can be divided into 100×100 search candidate regions; assuming the resolution of comparison image I' is also 100×100, comparison image I' can be divided into 100×100 regions. For each search candidate region in input image I, 100×100 regions in comparison image I' need to be replaced one by one. Therefore, for each key region found, 100×100×100×100 (i.e., 10 to the power of 8) replacements are required. When the number of key regions is large, the number of replacements will increase exponentially, leading to an excessive number of replacements in the end. In this context, a key region refers to a portion of the input image. Replacing a key region in the input image with a portion of the comparison image increases the probability that the input image will be predicted to belong to the same category as the comparison image. When all key regions in the input image are replaced with regions from the comparison image, the replaced input image can then be predicted to belong to the same category as the comparison image.

[0136] To ensure the accuracy of counterfactual interpretation results, current methods typically involve region-by-region replacement on high-resolution images. Due to the high image resolution, this requires replacing a large number of regions, leading to low processing efficiency and often a long processing time before outputting results.

[0137] In view of this, embodiments of this application provide an image processing method that acquires multiple feature maps of different resolutions from an input image, searches for key regions on the lower-resolution feature map to change the classification result, determines the mapping region of the key region on the lower-resolution feature map to the higher-resolution feature map, and further searches for key regions on this mapping region, ultimately obtaining the region on the image that needs to be replaced. By first searching for key regions on the low-resolution feature map and then searching for key regions within the corresponding region on the high-resolution feature map, the area to be searched on the high-resolution feature map is effectively narrowed down, improving efficiency and reducing the time required to output the result.

[0138] The image processing method provided in this application can be applied to a terminal. Exemplarily, the terminal may be a digital camera, surveillance camera, mobile phone, personal computer (PC), laptop computer, server, tablet computer, smart TV, mobile internet device (MID), wearable device, virtual reality (VR) device, augmented reality (AR) device, wireless terminal in industrial control, wireless terminal in self-driving, wireless terminal in remote medical surgery, wireless terminal in smart grid, wireless terminal in transportation safety, wireless terminal in smart city, wireless terminal in smart home, etc. For ease of description, the image processing method provided in this application will be described below using the application of the image processing method to a terminal as an example.

[0139] It should be understood that the images in the embodiments of this application can be static images (or static pictures) or dynamic images (or dynamic pictures), such as RGB images, black and white images, or grayscale images. For ease of description, in the following embodiments, static images or dynamic images will be uniformly referred to as images.

[0140] To facilitate understanding, the following will describe the application scenarios of the image processing method provided in this application embodiment. In the image processing method provided in this application embodiment, an output image can be generated by inputting an original image and a comparison image. This output image is obtained by replacing a portion of the original image with a portion of the comparison image, and the classification result of the output image is the same as the classification result of the comparison image. The image processing method provided in this application embodiment can be applied to fields such as educational products, computer program development, medical care, and transportation. This embodiment does not limit the application scenarios of this image processing method.

[0141] Specifically, image processing methods based on inverse real-time interpretation can provide explanations that align with human cognition, helping users understand the input factors influencing model decisions and how to modify these factors to make the model's decisions more closely resemble human expectations. By comparing the different interpretations provided by this image processing method on the same data and different models, users can identify potential problems in the model and compare its performance. This image processing method can also be applied to real-world scenarios involving fine-grained classification problems, such as assisting doctors in medical diagnoses and helping beginners quickly understand the differences between species within the same class but different species.

[0142] For example, please refer to Figure 6(a), which is a schematic diagram of an application scenario in an educational product provided by an embodiment of this application. As shown in Figure 6(a), children who are new to birds can select a set of data to be explained. This set of data to be explained includes: images of birds they do not recognize (i.e., the original images mentioned above) and images of birds they already recognize (i.e., the comparison images mentioned above).

[0143] Next, the prepared bird classification model and the data to be explained are input into the counterfactual interpretation module. During the input process, the bird classification model can be encapsulated, for example, by adding an API call, allowing the counterfactual interpretation module to be integrated with it. After the data to be explained is input into the counterfactual interpretation module, it calculates and outputs the result: the image obtained after replacing the image of the unfamiliar bird. In this way, children can compare the differences between unfamiliar and familiar birds by observing the output, thus learning the distinguishing features of unfamiliar birds.

[0144] For example, please refer to Figure 6(b), which is a schematic diagram of an application scenario in AI program development provided by an embodiment of this application. As shown in Figure 6(b), the AI ​​developer prepares multiple image classification models in advance, which are the models that the AI ​​developer needs to compare. Next, the AI ​​developer selects the image data that needs to be interpreted.

[0145] Then, the AI ​​developer inputs one of the prepared image classification models and the data to be explained into the counterfactual interpretation module. After the data to be explained is input into the counterfactual interpretation module, it calculates and outputs the result, which is the image obtained after replacing the aforementioned image data. After obtaining a set of output data corresponding to one image classification model, the AI ​​developer inputs another image classification model into the counterfactual interpretation model, and repeats the input of the data to be explained, obtaining the output data of the counterfactual interpretation model. This process is repeated until the output data corresponding to all image classification models is obtained. In this way, the AI ​​developer can determine the region of interest of different image classification models based on different output data, thereby determining which image classification model is more reliable.

[0146] For example, please refer to Figure 6(c), which is a schematic diagram of another application scenario in AI program development provided by an embodiment of this application. As shown in Figure 6(c), the AI ​​developer prepares an image classification model to be detected in advance and selects image data to be interpreted.

[0147] Then, the AI ​​developer inputs the prepared image classification model and the data to be explained into the counterfactual interpretation module. After the data to be explained is input into the counterfactual interpretation module, the module calculates and outputs a result, which is the image obtained after replacing the aforementioned image data. After obtaining a set of output data corresponding to the image classification model, the AI ​​developer can retrain and adjust the image classification model based on the output data. After obtaining the adjusted image classification model, the AI ​​developer re-inputs the adjusted image classification model and the data to be explained into the counterfactual interpretation module to obtain new output data. This process is repeated until the AI ​​developer is satisfied with the classification accuracy of the adjusted image classification model. In this way, the AI ​​developer can continuously adjust the image classification model based on the output data of the counterfactual interpretation module, thereby obtaining an image classification model with high classification accuracy.

[0148] Similarly, in the medical field, by inputting an image to be diagnosed and a comparison image, doctors can learn and determine which locations in the image to be diagnosed are key locations that influence the type of diagnosis. This embodiment will not elaborate on the specific procedures for applying this image processing method in various fields.

[0149] Please refer to Figure 7 , Figure 7 This is a schematic flowchart illustrating an image processing method provided in an embodiment of this application. Figure 7 As shown, the image processing method includes the following steps 701-705.

[0150] Step 701: Process the first image through the first network to obtain multiple feature maps with different resolutions.

[0151] In this embodiment, the first network can be a neural network, such as a convolutional neural network. The first network may include multiple feature extraction layers connected sequentially. After the first image is input into the first network, the multiple feature extraction layers perform feature processing on the first image, outputting feature maps of different resolutions.

[0152] The first image can be an image that requires counterfactual interpretation. The result of performing counterfactual interpretation on the first image is: which regions in the first image, when replaced, will change the category of the first image. For example, the first image can be an image of a bird that the child does not recognize, as shown in the example corresponding to Figure 6(a) above.

[0153] The resolution of the first image can be greater than or equal to that of the plurality of feature maps. For example, assuming the resolution of the first image is 1000×1000, the resolutions of the multiple feature maps obtained after processing the first image can be 250×250, 50×50, and 10×10, respectively. Typically, the resolution of the feature maps gradually decreases during the feature map extraction process of the first network. Therefore, in this case, the higher the resolution of the feature maps, the closer the feature maps are to the first image.

[0154] Step 702: Based on the second image, search the first feature map through the second network to obtain the first region in the first feature map.

[0155] In this embodiment, the second image is a comparison image of the first image. The classification results of the first image and the second image are different, that is, the category to which the first image belongs is different from the category to which the second image belongs. For example, assuming that both the first image and the second image are images of birds, after inputting the first image and the second image into the classification network respectively, the classification result of the first image is crested puffin, and the classification result of the second image is red-faced cormorant.

[0156] After obtaining multiple feature maps corresponding to the first image, one feature map (i.e., the first feature map mentioned above) can be selected from the multiple feature maps, for example, a feature map with a lower resolution can be selected. Then, based on the second image, a second network searches for key regions in the selected first feature map. The second network may specifically include a classification network that can predict the category to which the feature map belongs based on the feature map, thereby obtaining a classification result.

[0157] The search in the first feature map refers to replacing each region in the first feature map with a corresponding region in the second image. This process determines which region in the first feature map, when replaced, increases the probability that the classification network will predict the category of the second image. For example, assuming the initial probability of the classification network predicting the category of the second image for the first feature map is 0.1, after some regions in the first feature map are replaced with corresponding regions in the second image, the probability of the classification network predicting the category of the second image for the replaced first feature map becomes 0.3. The increased probability of the replaced first feature map predicting the category of the second image is 0.1. Therefore, the replaced region in the first feature map can be identified as the first region, which is used to change the classification result of the first feature map.

[0158] Optionally, since this embodiment performs a key region search on the first feature map, and the resolution of the second image is usually different from that of the first feature map, when replacing the region in the first feature map, the region in the first feature map can be replaced with the region in the feature map corresponding to the second image.

[0159] For example, the search of the region in the first feature map may specifically include: processing the second image through the first network to obtain a third feature map, the resolution of which is the same as that of the first feature map; replacing a portion of the first feature map with a portion of the third feature map, so that the classification result of the first feature map after the partial replacement is the same as the classification result of the second image; and determining the replaced region in the first feature map after the partial replacement as the first region. That is, the region corresponding to the second image may refer to a portion of the third feature map corresponding to the second image, the resolution of which is the same as that of the first feature map.

[0160] Generally, simply replacing a region in the first feature map will not change the classification result of the first feature map to be the same as the classification result of the second image. Therefore, during the search for regions in the first feature map, regions in the first feature map can be searched one by one, and after a region is found, the searched region in the first feature map is replaced. This process is repeated until the classification result of the first feature map after replacing one or more regions changes to the classification result of the second image.

[0161] For example, during the first key region search, each region in the first feature map is sequentially replaced with regions in the third feature map, and the classification result of the first feature map after the region replacement is predicted by the second network. Based on the classification result predicted by the second network, the probability increase of each first feature map after the region replacement being predicted as belonging to the category of the second image is obtained, thereby determining the region that maximizes the probability of the first feature map being predicted as belonging to the category of the second image, and this region is determined as the region searched in the first key region search process. After obtaining the key region in the first search, the key region in the first feature map is replaced with the corresponding region in the third feature map, resulting in the first feature map after one region replacement.

[0162] Then, based on the first feature map after replacing the key regions once, a second key region search is performed to obtain new key regions. Based on the first feature map after replacing the key regions once, the new key regions found during the second key region search are replaced to obtain the first feature map after replacing the key regions twice. This process is repeated until the first feature map after replacing the key regions N times is predicted by the second network to belong to the category of the second image.

[0163] In other words, in this embodiment, the first region may include one or more key regions. After replacing the first region in the first feature map with the corresponding region in the second image, a third image is obtained, which is the first feature map after replacing the first region. The classification result of this third image is the same as the classification result of the second image.

[0164] Step 703: Obtain the mapping region of the first region in the second feature map, wherein the content of the first region is related to the mapping region.

[0165] The second feature map is extracted by other feature extraction layers in the first network, and its resolution is greater than that of the first feature map. The content of the first region is related to the mapped region. In simple terms, the second feature extraction layer in the first network outputs the second feature map, which is then input into the feature extraction layer that outputs the first feature map. The feature extraction layer that outputs the first feature map processes the second feature map and outputs the first feature map. The first region in the first feature map is obtained by the feature extraction layer that outputs the first feature map processing the mapped region in the second feature map.

[0166] In other words, the mapped region is actually the receptive field of the first region in the second feature map. This receptive field can be obtained based on the kernel size and stride of the feature extraction layer that extracts the first feature map, as well as the first region.

[0167] For example, obtaining the mapping region of the first region in the second feature map may specifically include: obtaining the kernel size and stride of the feature extraction layer used to extract the first feature map; and determining the mapping region in the second feature map based on the first region and the kernel size and stride of the feature extraction layer.

[0168] Referring to Figure 8(a), which is a schematic diagram of a calculation of the receptive field provided by an embodiment of this application. As shown in Figure 8(a), Figure 8(a) includes the feature map extracted by the (i-1)th feature extraction layer (hereinafter referred to as feature map i-1), feature map i, and feature map i+1. The value "5" in feature map i+1 is calculated using the 2×2 region enclosed in feature map i. Therefore, the 2×2 region enclosed in feature map i can be called the receptive field of the region containing the value "5" in feature map i+1. Similarly, each region with a value of "3" in feature map i is calculated using the 3×3 region in feature map i-1, i.e., the kernel size is 3. Furthermore, the stride from feature map i-1 to feature map i is 2, and the 5×5 region in feature map i-1 is the receptive field of the 2×2 region enclosed in feature map i. In other words, the receptive field of a specific region in the feature map in the previous feature map is obtained based on the kernel size and stride.

[0169] Specifically, the mapping region in the second feature map can be obtained using the following formula 1.

[0170] S=(S1-1)*stride +ksize Formula 1

[0171] Where S is the size of the mapped region in the second feature map, S1 is the size of the first region, stride is the size of the convolution kernel of the feature extraction layer used to extract the first feature map, and ksize is the stride of the feature extraction layer used to extract the first feature map.

[0172] In one possible implementation, the first feature map can be the feature map with the lowest resolution among the plurality of feature maps. Among the plurality of feature maps, the first feature map has the fewest regions to be searched. Therefore, compared to the other feature maps among the plurality of feature maps, the regions in the first feature map can be searched in the least amount of time.

[0173] Optionally, the second feature map can be the feature map among the plurality of feature maps whose resolution is closest to that of the first feature map. That is, the feature extraction layer used to extract the second feature map and the feature extraction layer used to extract the first feature map can be two adjacent feature extraction layers. For example, if the first feature map is the feature map with the lowest resolution among the plurality of feature maps, then the feature extraction layer used to extract the first feature map is the last feature extraction layer of the first network, and the feature extraction layer used to extract the second feature map is the second-to-last feature extraction layer in the first network.

[0174] Optionally, the resolution of the second feature map can be a first preset resolution, meaning the second feature map is extracted by a feature extraction layer capable of obtaining a feature map at the first preset resolution. For example, assuming the resolution of the first image is 1000×1000, the resolution of the first feature map can be 10×10, and the resolution of the second feature map can be 250×250. This first preset resolution can, for example, be a resolution specified by the user. In practical applications, the user adjusts the first preset resolution according to actual needs. For example, when the user wants to obtain more accurate output results, the user can adjust the first preset resolution to a larger resolution; when the user wants to obtain output results faster, the user can adjust the first preset resolution to a smaller resolution.

[0175] When the resolution of the second feature map is a first preset resolution, the feature extraction layer used to extract the second feature map and the feature extraction layer used to extract the first feature map can be two adjacent feature extraction layers, or the feature extraction layer used to extract the second feature map and the feature extraction layer used to extract the first feature map can be separated by one or more other feature extraction layers.

[0176] Step 704: Based on the second image, search the mapped region through the second network to obtain the second region in the second feature map.

[0177] The second region may include one or more regions. Replacing the second region in the second feature map with the corresponding region in the second image yields a second feature map with the replaced second region. The classification result of this second feature map with the replaced second region is the same as the classification result of the second image. The corresponding region in the second image may refer to a portion of the fourth feature map corresponding to the second image, and the resolution of the fourth feature map corresponding to the second image is the same as the resolution of the second feature map.

[0178] The process of performing a region search in the mapped region of the second feature map using the second network is similar to the process of performing a region search in the first feature map using the second network, as described in step 702 above, and will not be repeated here. It is worth noting that in step 702, a region search was performed on all regions in the first feature map. However, in this step, the region search is performed in the mapped region of the second feature map, thus narrowing the search area. It no longer performs a region search on all regions in the second feature map, thereby saving computational resources and significantly reducing computation time.

[0179] Step 705: Based on the second region, obtain the region in the first image that corresponds to the second region.

[0180] Since the obtained second region is a region in the second feature map, the second region in the second feature map can be converted into a region corresponding to the first image to output a first image marked with the corresponding region.

[0181] Optionally, obtaining the region in the first image corresponding to the second region based on the second region specifically includes: obtaining the resolution of the second feature map and the resolution of the first image. The region in the first image corresponding to the second region is determined based on the position of the second region in the second feature map, the resolution of the second feature map, and the resolution of the first image.

[0182] For example, please refer to Figure 8(b), which is a schematic diagram of converting a second feature map into a first image according to an embodiment of this application. As shown in Figure 8(b), the resolution of the second feature map is 4×4, and the resolution of the first image is 8×8. In the second feature map, the three regions in the lower right corner are determined as the second region. Based on the position of the second region in the second feature map, and the resolutions of the second feature map and the first image, the corresponding region of the second region can be determined in the first image. This corresponding region is the 12 regions in the lower right corner of the first image.

[0183] In this embodiment, multiple feature maps of different resolutions of the input image are acquired. A region search is performed on the lower-resolution feature map, and then the mapping region of the region on the lower-resolution feature map is determined on the higher-resolution feature map. Further searching is conducted within this mapping region to ultimately obtain the region in the image that needs to be replaced. By first searching for the region on the low-resolution feature map and then searching for the corresponding region on the high-resolution feature map, the region to be searched on the high-resolution feature map is effectively narrowed down, improving efficiency and reducing the time required to output the results.

[0184] To facilitate understanding, the image processing method described in the embodiments of this application will be introduced below with specific examples. Please refer to... Figure 9 , Figure 9 This is a schematic diagram of a search area provided in an embodiment of this application.

[0185] like Figure 9 As shown, firstly, the first image is input into the first network, which performs feature extraction on the first image to obtain a first feature map and a second feature map output by the first network. Then, based on the second image, the second network performs a region search on all regions of the first feature map to obtain the first region on the first feature map. After obtaining the first region, the mapping region of the first region in the first feature map on the second feature map is calculated, and the region search continues in the mapping region. Finally, after obtaining the second region in the second feature map, the second region in the second feature map is converted into the corresponding region of the first image to output the first image marked with the corresponding region.

[0186] In one possible embodiment, in order to maximize the accuracy of the output results, region search can be performed on each of the multiple feature maps obtained, and the search range of each feature map is the mapping area of ​​the key region already searched in the previous feature map.

[0187] For example, the image processing method described above may further include: repeatedly performing the region search step until a termination condition is met to obtain a target region, which is a region in the feature map of the last search. The region in the first image corresponding to the target region is then output.

[0188] Specifically, the region search step includes: obtaining the mapping region of the region in the i-th feature map to the (i+1)-th feature map, wherein the region in the i-th feature map is used to change the classification result of the i-th feature map; based on the second image, searching in the mapping region of the (i+1)-th feature map through the second network to obtain the region in the (i+1)-th feature map, wherein the region in the (i+1)-th feature map is used to change the classification result of the (i+1)-th feature map; and incrementing the value of i by 1.

[0189] Wherein, the i-th feature map and the (i+1)-th feature map are both feature maps from multiple feature maps extracted by the first network, and the resolution of the i-th feature map is less than the resolution of the (i+1)-th feature map. i is greater than or equal to 2, and when i equals 2, the i-th feature map is the same as the second feature map. The target region is the region in the (i+1)-th feature map that satisfies the termination condition.

[0190] For example, assuming the initial value of i is 2, then the second feature map is the second feature map mentioned above. Therefore, the region in the second feature map (i.e., the second region mentioned above) is mapped to the third feature map; based on the second image, the second network searches for the region in the mapped region of the third feature map to obtain the region in the third feature map. Next, i is incremented by 1, and its value becomes 3. Then, the region in the third feature map is mapped to the fourth feature map; based on the second image, the second network searches for the region in the mapped region of the fourth feature map to obtain the region in the fourth feature map. This process continues, searching for regions in multiple feature maps one by one until the termination condition is met.

[0191] In simple terms, assume that the multiple feature maps obtained by the first network are extracted by multiple feature extraction layers connected sequentially within the first network. The first feature map is extracted by the last feature extraction layer in the first network, and the second feature map is extracted by the second-to-last feature extraction layer. Therefore, after obtaining the second region in the second feature map, the mapping region of the second region in the feature map extracted by the third-to-last feature extraction layer in the first network is determined, and the search for regions continues within this mapping region. After obtaining the region in the feature map extracted by the third-to-last feature extraction layer, the mapping region in the feature map extracted by the fourth-to-last feature extraction layer is determined based on this region, and the search for regions continues. This process continues, each time a region is obtained in a feature map, the corresponding mapping region is determined in the feature map extracted by the previous feature extraction layer, and the search for regions continues within this mapping region until the termination condition is met.

[0192] In this embodiment, there can be multiple termination conditions. In practical applications, the termination conditions can be adjusted according to actual needs.

[0193] In one possible implementation, the termination condition includes the resolution of the (i+1)th feature map reaching a second preset resolution. It should be understood that a higher resolution of the feature map used for the search region means less information is contained within a unit area of ​​the feature map, resulting in a more accurate search region. However, this also means a larger search area and more time is required. Therefore, in practical applications, a balance can be struck between the accuracy of the results and computational efficiency based on the user's needs to determine the final resolution of the searched feature map. That is, the second preset resolution can be a predefined resolution, set by the user or the network developer, such as 250×250 or 100×100. No limitation is placed on the second preset resolution here.

[0194] In another possible implementation, the termination condition includes a ratio of the area between the region in the (i+1)th feature map and the mapped region in the (i+1)th feature map being greater than a preset threshold. For example, the preset threshold could be a value such as 2 / 3, 3 / 4, or 4 / 5; this embodiment does not specifically limit the value of the preset threshold.

[0195] Understandably, during the region search process, the process of continuing to search for a region in the mapped region of the (i+1)th feature map after searching the region of the i-th feature map is actually to further narrow down the region's range to obtain a more accurate result. When the search finds that the area ratio between the region in the (i+1)th feature map and its mapped region is greater than a preset threshold, it can be considered that the region cannot be narrowed down by continuing to search for a region in the mapped region of the (i+1)th feature map, and therefore the region search can be terminated.

[0196] In practical applications, during the region search process in the (i+1)th feature map, if the area ratio between the replaced region and the mapped region in the (i+1)th feature map is greater than or equal to a preset threshold, and the classification result of the (i+1)th feature map after partial region replacement still does not change to the classification result of the second image, then it can be considered that the area ratio between the region in the (i+1)th feature map and the mapped region in the (i+1)th feature map is greater than the preset threshold. At this point, the region search for the (i+1)th feature map can be terminated, and the mapped region of the (i+1)th feature map can be used as the region of the (i+1)th feature map, thus outputting the region of the (i+1)th feature map.

[0197] Understandably, for an image to be classified, it usually includes other objects besides the target object. For example, a bird image might include branches, sky, or grass in addition to the bird. In this case, replacing the objects other than the bird will not change the classification result of the replaced image. In other words, for an image to be classified, replacing other objects besides the target object will not change the classification result.

[0198] Therefore, if objects other than the target object can be identified before performing a search of the region, the regions containing these objects can be locked, meaning that the search of the region will not be performed in these regions, thereby reducing the search range and improving search efficiency.

[0199] In this embodiment, the search range of the region can be reduced in a variety of ways.

[0200] In one possible implementation, an algorithm can be used to identify background regions in the input image, thereby excluding these regions from the search scope and reducing the search area.

[0201] For example, the step of searching the first feature map based on the second image using a second network includes: obtaining non-background regions in the first feature map, where the non-background regions are regions in the first feature map other than the background regions. For instance, an algorithm can be used to identify background and non-background regions in the first image, and these regions can be labeled accordingly. After obtaining multiple feature maps of the first image using the first network, the corresponding background and non-background regions in the feature maps can be determined based on the mapping relationship between the first image and the feature maps. Then, based on the second image, the second network searches the non-background regions in the first feature map. That is, background regions in the first feature map can be considered locked regions, and these locked regions are not searched during the region search process.

[0202] In another possible implementation, the user can specify areas that do not need to be searched, thereby excluding these user-specified areas from the search scope and reducing the search range. For example, when the image processing method provided in this embodiment is applied to an educational product, for an image of a bird that a child does not recognize (the bird image is used as the first image input to the first network), the child can specify a familiar feature region in the bird image, thereby locking in these feature regions that are not included in the search area.

[0203] For example, the search in the first feature map based on the second image via the second network includes: First, obtaining a marked region in the first image, which can be a region marked by the user to indicate a region where region search is not required. This marked region can also be referred to as a locked region indicated by the user. Then, determining a fourth region in the first feature map corresponding to the marked region. Since the marked region is marked in the first image, and the region search process is performed on the first feature map, the region corresponding to the marked region in the first image in the first feature map can be determined, i.e., the aforementioned first region. Specifically, the process of determining the first region in the first feature map can be based on feature extraction performed by the first network to determine the region in the first feature map related to the marked region in the first image, and then defining this region as the first region. Finally, based on the second image, searching in a third region in the first feature map via the second network, where the third region is the region in the first feature map other than the fourth region. After obtaining the fourth region, the region in the first feature map other than the fourth region can be determined as the third region, and a key region search can be performed on the second region to narrow down the search range.

[0204] It is understandable that the two implementation methods described above can be used in combination. For example, after the background region in the first image is identified by the algorithm, this background region is designated as the default locked region. The user can then continue to mark other unlocked regions on the first image to obtain the user-selected locked region. In this way, the first image includes both the default locked region and the user-selected locked region. When performing a region search on the first feature map, the search is not performed within the regions corresponding to the default locked region and the user-selected locked region.

[0205] For example, please refer to Figure 10(a), which is a schematic diagram of a region search based on a locked region provided in an embodiment of this application. As shown in Figure 10(a), the first image is a bird image. After the background recognition algorithm identifies the first image, most of the background regions in the first image are identified, and the identified background regions are determined as default locked regions. On the first image with the default locked regions determined, the user further marks other regions, such as the bird's back and chest, and the regions marked by the user are determined as user-selected locked regions. In this way, the first image includes both the default locked regions and the user-selected locked regions. When performing a region search on the first feature map corresponding to the first image, the region search is not performed within the regions corresponding to the default locked regions and the user-selected locked regions. As can be seen from the output result diagram on the right side of Figure 10(a), the regions corresponding to the regions in the first image are all outside the default locked regions and the user-selected locked regions.

[0206] For the same image, the output result changes as the locked region changes. For example, refer to Figure 10(b), which is a comparative schematic diagram of region search based on a locked region according to an embodiment of this application. As shown in Figure 10(b), the marked regions in the output result image are different when the locked region in the first image is different.

[0207] In addition to the methods described above for narrowing the search range of the feature map by identifying the background region in the first image and obtaining the marked region in the first image, other methods can also be used to narrow the search range of the feature map. This embodiment does not specifically limit these methods.

[0208] Understandably, the second region on the second feature map can include multiple regions, and each region contributes differently to changing the classification result of the replaced second feature map to the classification result of the second image. For regions containing more features or more obvious features, replacing these regions with the second feature map significantly increases the probability of them being predicted as the classification result of the second image. For regions containing fewer features or less obvious features, replacing these regions with the second feature map slightly increases the probability of them being predicted as the classification result of the second image.

[0209] Based on this, in one possible embodiment, regions can be sorted and displayed according to their contribution to the classification result of the second feature map, so as to highlight the importance of different regions.

[0210] For example, the second region of the second feature map may include multiple sub-regions. The step of obtaining the region in the first image corresponding to the second region based on the second region specifically includes: first, for each of the multiple sub-regions in the second feature map, obtaining the probability corresponding to each sub-region, where the probability is the increase in the probability of the second feature map being predicted as the category of the second image after replacing the sub-region. For example, for any sub-region in the second feature map, assuming the probability of the second feature map being predicted as the category of the second image before replacing the sub-region is s, and the probability of the second feature map being predicted as the category of the second image after replacing the sub-region is n, then the probability corresponding to the sub-region is ns.

[0211] Then, the probabilities corresponding to each sub-region are sorted in descending order to obtain the sorting result. For example, assuming the probability corresponding to the first sub-region is 0.3, the probability corresponding to the second sub-region is 0.1, and the probability corresponding to the third sub-region is 0.2, the following sorting result can be obtained: first sub-region > third sub-region > second sub-region.

[0212] Based on the sorting result and the plurality of sub-regions, a plurality of regions corresponding to the plurality of sub-regions are obtained in the first image. The plurality of regions have different marking methods, and the marking methods are related to the sorting result. The different marking methods may include marking with marking boxes of different colors, marking with marking boxes of different shapes, or marking with marking boxes of different thicknesses. This embodiment does not limit the marking methods.

[0213] For example, as shown in Figure 10(a), the most significant differences between the birds in the first and second images lie in the crest, beak, eyes, and chest areas, with the order of difference between these features being crest, beak, eyes, and chest. That is, the crests of the two birds are the most different, followed by the beak, then the eyes, and finally the chest. After ranking these different areas according to their importance, different colored boxes can be used to mark these areas based on the ranking result. For example, a red box can be used to mark the area containing the crest; a yellow box can be used to mark the area containing the beak; a blue box can be used to mark the area containing the eyes; and a green box can be used to mark the area containing the beak.

[0214] To facilitate understanding, the image processing method provided in this application embodiment will be described in detail below with specific examples. For example, please refer to... Figure 11 and Figure 12 , Figure 11A schematic flowchart of an image processing method provided in an embodiment of this application; Figure 12 This is a schematic diagram illustrating an example of searching key regions provided in an embodiment of this application. For example... Figure 11 As shown, the process for searching key areas includes the following steps S1-S9.

[0215] S1, the original image (i.e., the first image mentioned above) and the comparison image (i.e., the second image mentioned above) are input into the counterfactual explanation network, which includes the first network and the second network mentioned above.

[0216] In addition, the original image may include locked areas, such as areas locked by default or areas locked by the user.

[0217] S2, multiple feature maps corresponding to the original image are extracted through the counterfactual explanation network. These feature maps have different resolutions. Furthermore, multiple feature maps corresponding to the comparison image can also be extracted through the network structure used to extract the original image within the counterfactual explanation network. These feature maps also have different resolutions. In short, for each feature map in the multiple feature maps corresponding to the original image, there is a feature map in the comparison image with the same resolution.

[0218] S3, in the multiple feature maps corresponding to the original image, search for key regions (such as...) in the last layer of feature maps. Figure 12 The key region 1 shown refers to the last feature map extracted by the last feature extraction layer. This last feature map is the image with the lowest resolution among the multiple feature maps corresponding to the original image. For the multiple feature extraction layers used to extract feature maps from the original image, the resolution of the feature maps extracted by the feature extraction layers gradually increases from the last feature extraction layer to the first feature extraction layer. The specific search process can be found in step 702 above, and will not be repeated here.

[0219] When the original image includes locked regions, the search for key regions is not performed within the locked regions when searching for key regions in the last layer feature map.

[0220] S4, after obtaining the key regions of the last layer feature map, determine the mapping regions in the penultimate layer feature map based on the key regions of the last layer feature map (e.g., ...). Figure 12 The mapping region 1 shown is the key region of this last feature map, determined based on the mapping region in the penultimate feature map. This penultimate feature map refers to the feature map extracted by the penultimate feature extraction layer.

[0221] S5, continue searching for key regions (such as...) in the mapping region of the penultimate feature map. Figure 12 The key area shown is 2). For the specific search process, please refer to step 704 above, which will not be repeated here.

[0222] S6. Determine whether the termination condition is met. The termination condition may be, for example, that the resolution of the current feature map reaches a second preset resolution, or that the area ratio between the key region and the mapped region in the current feature map is greater than a preset threshold. If the termination condition is met, proceed to step S7; otherwise, proceed to step S8.

[0223] S7 outputs the region in the original image corresponding to the key region. When the termination condition is met, based on the key region in the final obtained feature map, the region in the original image corresponding to the key region is obtained and output.

[0224] S8, if the termination condition is not met, continue to determine the mapping region in the next layer feature map (e.g., Figure 12 The mapping region shown is 2). For example, the mapping region is determined in the third-to-last feature map, the fourth-to-last feature map, or the fifth-to-last feature map.

[0225] S9, after determining the mapping region of the feature map, then search for key regions (such as...) within the mapping region of the feature map. Figure 12 The key area 3 shown in the figure is then processed, and the process proceeds to step S6.

[0226] Please refer to Figure 13 , Figure 13 This is a schematic diagram illustrating a layer-by-layer search for key regions, provided as an embodiment of this application. For example... Figure 13 As shown, the first network includes multiple convolutional layers, multiple pooling layers, and multiple fully connected layers. After the first image is input into the first network, each convolutional layer and each pooling layer in the first network outputs a different feature map. Therefore, during the key region search process, pooling layer 4 is used as the last feature extraction layer, and key region search is performed on the feature map extracted by pooling layer 4. After obtaining the key regions in the feature map extracted by pooling layer 4, the mapping region of the key regions in the feature map extracted by pooling layer 4 is determined in convolutional layers 4-3, and the key region search continues to be performed in the mapping region. This process is repeated layer by layer, starting from pooling layer 4, until the termination condition is met.

[0227] Please refer to Figure 14 , Figure 14 This is a comparative schematic diagram of a key area provided in an embodiment of this application. For example... Figure 14As shown, after searching for key regions on feature maps of different resolutions, the size and position of the corresponding regions in the original image may differ when these key regions are converted to their corresponding regions in the original image. For example, when searching for key regions in the feature map extracted by pooling layer 4 and converting them to their corresponding regions in the original image, the corresponding regions in the original image are larger (i.e., the regions marked by the bounding boxes in the original image are larger), and these regions contain not only distinguishing features but also other non-discriminating features. When searching for key regions in the feature map extracted by pooling layer 2 and converting them to their corresponding regions in the original image, the corresponding regions in the original image are smaller (i.e., the regions marked by the bounding boxes in the original image are smaller). The resolution of the feature map extracted by pooling layer 4 is lower than that of the feature map extracted by pooling layer 2. In other words, searching for key regions on feature maps with higher resolutions yields more accurate output results.

[0228] Furthermore, compared to performing key region searches directly on feature maps of a specific resolution, searching key regions layer by layer in the mapped region as described in the above embodiments can effectively improve computation speed and consume less memory, allowing users to obtain output results with less computational resources and time.

[0229] For example, suppose t1 is the time to synthesize a feature map (i.e., the time required to replace a region in the feature map), and t2 is the time to infer from a feature map, i.e., the time to obtain the classification result of a feature map.

[0230] Therefore, Formula 2 can be used to calculate the time required to search for the key region on the feature map.

[0231] t= N*R^2*t1 + N*R^2 / B *t2 Formula 2

[0232] Where t is the time required to perform a key region search on a feature map, N is the number of key regions found on the feature map, t1 is the time to synthesize a feature map, t2 is the time to infer a feature map, R is the size of the feature map, R^2 represents the square of R, and B is the number of feature maps for batch inference.

[0233] by Figure 14 For example, assume the feature map extracted by pooling layer 4 is 7*7 in size, and the feature map extracted by convolutional layer 4-3 is 14*14 in size. Assume the number of key regions extracted from pooling layer 4 is N1, the number of key regions extracted from convolutional layer 4-3 is N2, and the receptive field size from pooling layer 4 to convolutional layer 4-3 is 2*2.

[0234] Then, the time required to search for the key region in the feature map of the pooling layer 4 is N1*(7*7)^2*t1 + N1*(7*7)^2 / B*t2.

[0235] After the key region is searched in the feature map of the pooling layer 4, the time required to search for the key region in the mapped region of the feature map of the convolutional layer 4-3 is N2*(2*2)^2*t1 + N2*(2*2)^2 / B*t2.

[0236] That is, the time required to search for the key region in the feature map of the convolutional layer 4-3 based on the method provided in this embodiment is: N1*(7*7)^2*t1 + N2*(2*2)^2*t1 + N1*(7*7)^2 / B*t2 + N2*(2*2)^2 / B*t2.

[0237] If the key region is directly searched in the feature map of the convolutional layer 4-3, the time required is: N2*(14*14)^2*t1 + N2*(14*14) / B*t2.

[0238] Generally, since the time required for batch processing of the feature map is less, the inference time of the feature map can usually be ignored. Then, the main factor affecting the total time spent on searching for the key region is the time required to replace the feature map.

[0239] In most cases, the numerical difference between N1 and N2 is not large, so N1*(7*7)^2*t1 + N2*(2*2)^2*t1 < N2*(14*14)^2*t1. That is to say, compared with directly searching for the key region on the specified feature map, based on the method provided in the embodiment of the present application, searching for the key region layer by layer in the mapped region can effectively reduce the calculation time.

[0240] Reference can be made to Figure 15 , Figure 15 which is an image processing device provided in the embodiment of the present application. The image processing device includes an acquisition unit 1501 and a processing unit 1502.

[0241] The processing unit 1502 is configured to process the first image through a first network to obtain multiple feature maps, the multiple feature maps including a first feature map and a second feature map, wherein the resolution of the second feature map is greater than the resolution of the first feature map; the processing unit 1502 is further configured to search in the first feature map based on the second image through a second network to obtain a first region in the first feature map, wherein the first region is used to change the classification result of the first feature map, the classification result of the first image is different from that of the second image, and the classification result of the third image is the same as that of the second image, wherein the third image is the first feature map after replacing the first region with the corresponding region in the second image; the acquisition unit 1501 is configured to acquire the mapping region of the first region in the second feature map, wherein the content of the first region is related to the mapping region; the processing unit 1502 is further configured to search in the mapping region based on the second image through the second network to obtain a second region in the second feature map, wherein the second region is used to change the classification result of the first feature map; the processing unit 1502 is further configured to obtain a region in the first image corresponding to the second region based on the second region.

[0242] In one possible implementation, the first feature map is the feature map with the smallest resolution among the plurality of feature maps.

[0243] In one possible implementation, the second feature map is the feature map among the plurality of feature maps whose resolution is closest to that of the first feature map; or, the resolution of the second feature map is a first preset resolution; or, the second feature map and the first feature map are further separated by one or more feature maps among the plurality of feature maps, the resolution of the one or more feature maps being greater than the resolution of the first feature map and less than the resolution of the second feature map.

[0244] In one possible implementation, the acquisition unit 1501 is configured to: acquire the kernel size and stride of the feature extraction layer used to output the first feature map; and determine the mapping region in the second feature map based on the first region and the kernel size and stride of the feature extraction layer.

[0245] In one possible implementation, the processing unit 1502 is further configured to: repeatedly execute the region search step until a termination condition is met to obtain a target region; output the region in the first image corresponding to the target region; the region search step includes: obtaining the mapping region of the region in the i-th feature map in the (i+1)-th feature map, the region in the i-th feature map being used to change the classification result of the i-th feature map; based on the second image, performing a region search in the mapping region of the (i+1)-th feature map through the second network to obtain the region in the (i+1)-th feature map, the region in the (i+1)-th feature map being used to change the classification result of the (i+1)-th feature map; incrementing the value of i by 1; wherein, the target region is the region in the (i+1)-th feature map when the termination condition is met, the resolution of the i-th feature map is less than the resolution of the (i+1)-th feature map, i is greater than or equal to 2, and when i is equal to 2, the i-th feature map is the second feature map.

[0246] In one possible implementation, the termination condition includes the resolution of the (i+1)th feature map reaching a second preset resolution.

[0247] In one possible implementation, the termination condition includes a ratio of the area between the region in the (i+1)th feature map and the mapped region in the (i+1)th feature map being greater than a preset threshold.

[0248] In one possible implementation, the processing unit 1502 is further configured to: process the second image through the first network to obtain a third feature map, the resolution of the third feature map being the same as the resolution of the first feature map; replace a portion of the first feature map with a portion of the third feature map, so that the classification result of the first feature map after replacing the portion of the region is the same as the classification result of the second image; and determine the replaced region in the first feature map after replacing the portion of the region as the first region.

[0249] In one possible implementation, the acquisition unit 1501 is further configured to acquire a non-background region in the first feature map, wherein the non-background region is a region in the first feature map other than the background region; the processing unit 1502 is further configured to search for the non-background region in the first feature map based on the second image using the second network.

[0250] In one possible implementation, the acquisition unit 1501 is further configured to acquire a marked region in the first image; the processing unit 1502 is further configured to search for a third region in the first feature map based on the second image using the second network, wherein the third region is a region in the first feature map other than the fourth region, and the fourth region is a region in the first feature map corresponding to the marked region.

[0251] In one possible implementation, the processing unit 1502 is further configured to determine the region in the first image corresponding to the second region based on the position of the second region in the second feature map, the resolution of the second feature map, and the resolution of the first image.

[0252] In one possible implementation, the second region includes multiple sub-regions; the processing unit 1502 is further configured to: obtain the probability corresponding to each sub-region in the multiple sub-regions, the probability being the increased probability when the sub-region is predicted as the category of the second image after the second feature map replaces it; sort the probabilities corresponding to each sub-region in descending order to obtain a sorting result; and obtain multiple regions in the first image corresponding to the multiple sub-regions based on the sorting result and the multiple sub-regions, the multiple regions having different labeling methods, the labeling methods being related to the sorting result.

[0253] The following describes an execution device provided in an embodiment of this application. Please refer to [link / reference]. Figure 16 , Figure 16 This is a schematic diagram of an execution device provided in an embodiment of this application. The execution device 1600 can specifically be a mobile phone, tablet, laptop, smart wearable device, server, etc., and is not limited thereto. The execution device 1600 may be equipped with... Figure 16 The data processing apparatus described in the corresponding embodiment is used to implement Figure 16 The corresponding embodiment describes the data processing function. Specifically, the execution device 1600 includes: a receiver 1601, a transmitter 1602, a processor 1603, and a memory 1604 (wherein the execution device 1600 may have one or more processors 1603). Figure 16 (Taking a processor as an example), processor 1603 may include application processor 16031 and communication processor 16032. In some embodiments of this application, receiver 1601, transmitter 1602, processor 1603 and memory 1604 may be connected via bus or other means.

[0254] Memory 1604 may include read-only memory and random access memory, and provides instructions and data to processor 1603. A portion of memory 1604 may also include non-volatile random access memory (NVRAM). Memory 1604 stores processor and operation instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0255] Processor 1603 controls the operation of the execution device. In specific applications, the various components of the execution device are coupled together through a bus system, which may include not only the data bus, but also power buses, control buses, and status signal buses. However, for clarity, all buses are referred to as the bus system in the diagram.

[0256] The methods disclosed in the embodiments of this application can be applied to or implemented by the processor 1603. The processor 1603 can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by the integrated logic circuits in the hardware of the processor 1603 or by instructions in software form. The processor 1603 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor, or a microcontroller, and may further include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The processor 1603 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory 1604. Processor 1603 reads the information in memory 1604 and, in conjunction with its hardware, completes the steps of the above method.

[0257] Receiver 1601 can be used to receive input digital or character information, and to generate signal inputs related to the settings and function control of the execution device. Transmitter 1602 can be used to output digital or character information through the first interface; transmitter 1602 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; transmitter 1602 may also include a display device such as a display screen.

[0258] In one embodiment of this application, the processor 1603 is configured to execute... Figure 7 The image processing method in the corresponding embodiment.

[0259] This application also provides a computer program product that, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0260] This application also provides a computer-readable storage medium storing a program for signal processing, which, when run on a computer, causes the computer to perform steps as performed by the aforementioned execution device, or causes the computer to perform steps as performed by the aforementioned training device.

[0261] The execution device, training device, or terminal device provided in this application embodiment can specifically be a chip. The chip includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit to cause the chip within the execution device to execute the image processing method described in the above embodiments, or to cause the chip within the training device to execute the image processing method described in the above embodiments. Optionally, the storage unit can be a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0262] For details, please refer to Figure 17 , Figure 17This is a schematic diagram of a chip provided in an embodiment of this application. The chip can be represented as a neural network processor (NPU) 1700. The NPU 1700 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core part of the NPU is the arithmetic circuit 1703, which is controlled by the controller 1704 to extract matrix data from the memory and perform multiplication operations.

[0263] In some implementations, the arithmetic circuit 1703 internally includes multiple processing engines (PEs). In some implementations, the arithmetic circuit 1703 is a two-dimensional pulsating array. The arithmetic circuit 1703 can also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1703 is a general-purpose matrix processor.

[0264] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 1702 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 1701 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 1708.

[0265] Unified memory 1706 is used to store input and output data. Weight data is directly transferred to weight memory 1702 via Direct Memory Access Controller (DMAC) 1705. Input data is also transferred to unified memory 1706 via DMAC.

[0266] BIU stands for Bus Interface Unit, which is used for interaction between the AXI bus and the DMAC and the Instruction Fetch Buffer (IFB) 1709.

[0267] The Bus Interface Unit (BIU) 1717 is used by the instruction fetch memory 1709 to fetch instructions from external memory, and also by the memory access controller 1705 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0268] The DMAC is mainly used to move input data from external memory DDR to unified memory 1706, or to weight data to weight memory 1702, or to input data to input memory 1701.

[0269] The vector computation unit 1707 includes multiple arithmetic processing units that, when needed, further process the output of the arithmetic circuit 1703, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is mainly used for computation in non-convolutional / fully connected layers of neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0270] In some implementations, the vector computation unit 1707 can store the processed output vector in the unified memory 1706. For example, the vector computation unit 1707 can apply a linear function, or a nonlinear function, to the output of the computation circuit 1703, such as performing linear interpolation on feature planes extracted by a convolutional layer, or, for example, accumulating a vector of values ​​to generate activation values. In some implementations, the vector computation unit 1707 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as an activation input to the computation circuit 1703, for example, for use in subsequent layers of the neural network.

[0271] The instruction fetch buffer 1709 connected to the controller 1704 is used to store the instructions used by the controller 1704;

[0272] Unified memory 1706, input memory 1701, weighted memory 1702, and instruction fetch memory 1709 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0273] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits used to control the execution of the above program.

[0274] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0275] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0276] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0277] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. An image processing method, characterized by, The method comprises: processing a first image through a first network to obtain a plurality of feature maps, the plurality of feature maps comprising a first feature map and a second feature map, the second feature map having a resolution greater than that of the first feature map; based on a second image, searching for a first region in the first feature map through a second network, the first region being a region that, when replaced by a corresponding region in the second image, causes a change in the classification result, wherein the first region is used to change the classification result of the first feature map, the classification result of the first image being different from that of the second image, the classification result of a third image being the same as that of the second image, the third image being the first feature map after replacing the first region with a corresponding region in the second image; obtaining a mapping region of the first region in the second feature map, the content of the first region being related to the mapping region; based on the second image, searching in the mapping region through the second network to obtain a second region in the second feature map, the second region being used to change the classification result of the second feature map; obtaining a region in the first image corresponding to the second region according to the second region.

2. The method of claim 1, wherein, The first feature map is the feature map with the smallest resolution in the plurality of feature maps.

3. The method according to claim 1 or 2, characterized in that, The second feature map is the feature map in the plurality of feature maps that is closest in resolution to the first feature map. Or, the resolution of the second feature map is a first preset resolution. Or, the second feature map and the first feature map are further separated by one or more feature maps in the plurality of feature maps, the resolution of the one or more feature maps being greater than that of the first feature map and less than that of the second feature map.

4. The method according to claim 1 or 2, characterized in that, The obtaining of the mapping region of the first region in the second feature map comprises: obtaining the convolution kernel size and step length of a feature extraction layer used to output the first feature map; determining the mapping region in the second feature map according to the first region and the convolution kernel size and step length of the feature extraction layer.

5. The method according to claim 1 or 2, characterized in that, The searching for a first region in the first feature map based on a second image through a second network comprises: processing the second image through the first network to obtain a third feature map, the resolution of the third feature map being the same as that of the first feature map; replacing part of the regions in the first feature map with part of the regions in the third feature map, so that the classification result of the first feature map after replacing the part of the regions is the same as that of the second image; determining the replaced region in the first feature map after replacing the part of the regions as the first region.

6. The method of claim 1 or 2, wherein, The searching in the first feature map based on a second image through a second network comprises: obtaining a marked region in the first image; based on the second image, searching in a third region in the first feature map through the second network, the third region being a region in the first feature map other than a fourth region, the fourth region being a region in the first feature map corresponding to the marked region.

7. The method according to claim 1 or 2, characterized in that, The obtaining, according to the second region, of a region in the first image corresponding to the second region comprises: determining, according to a position of the second region in the second feature map, a resolution of the second feature map, and a resolution of the first image, the region in the first image corresponding to the second region.

8. The method of claim 1 or 2, wherein, The second region comprises a plurality of sub-regions. The obtaining, according to the second region, of a region in the first image corresponding to the second region comprises: obtaining a probability corresponding to each of the plurality of sub-regions, the probability being a probability of the second feature map being predicted as a category of the second image after replacing the sub-region; sorting the probability corresponding to each of the plurality of sub-regions in descending order to obtain a sorting result; obtaining, according to the sorting result and the plurality of sub-regions, a plurality of regions in the first image corresponding to the plurality of sub-regions, the plurality of regions having different marking manners, the marking manners being related to the sorting result.

9. An image processing apparatus characterized by comprising: The image processing apparatus comprises a memory and a processor; the memory stores code, and the processor is configured to execute the code, and when the code is executed, the image processing apparatus performs the method according to any one of claims 1 to 8.

10. A computer storage medium, characterized in that, The computer storage medium stores instructions, and the instructions cause the computer to implement the method according to any one of claims 1 to 8 when executed by the computer.

11. A computer program product, characterised in that, The computer program product stores instructions, and the instructions cause the computer to implement the method according to any one of claims 1 to 8 when executed by the computer.

Images