Image processing method, apparatus, device, storage medium and computer program product

Abstract

The application discloses an image processing method, device and equipment, a storage medium and a computer program product. Related embodiments can be applied to artificial intelligence, intelligent transportation and automatic driving scenes. The image processing method comprises: training an image classification model based on a first sample image, a first category label, a second sample image and a second category label; and using a point loss function, a contrast loss function and a classification loss function to design a training loss function during the training process, thereby improving the classification accuracy of the image classification model. The value of the point loss function and the value of the contrast loss function are determined based on a first feature map set and a second feature map set, and the first feature map set and the second feature map set are obtained by identifying and processing the first sample image and the second sample image by the image classification model respectively. The value of the classification loss function is obtained based on the first feature map set, the second feature map set, the first category label and the second category label.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and in particular to an image processing method and device, equipment, a storage medium and a computer program product. BACKGROUND

[0002] An image classification task requires accurate classification of an image, that is, to accurately classify the image into a certain preset category among multiple preset categories. With the development of deep learning technology, image classification through an image classification model has become a common method for image classification. When training an image classification model, the global features of an image are usually analyzed and processed (that is, all image regions of the image are analyzed and processed), and the image is predicted to be a certain preset category according to the category features of the certain preset category possessed by the image to achieve classification of the image. However, when the feature region concentrated by the category features possessed by the image is small, the classification accuracy of the image classification model is low. Therefore, in the field of image processing, how to train an image classification model to improve the classification accuracy of the image classification model has become one of the hot issues of research. SUMMARY

[0003] The embodiments of the present application provide an image processing method, device, equipment, storage medium and computer program product, which can improve the image classification accuracy.

[0004] In one aspect, the embodiments of the present application provide an image processing method, comprising:

[0005] obtaining a training sample pair, the training sample pair comprising a first sample image, a second sample image, a first category label corresponding to the first sample image, and a second category label corresponding to the second sample image;

[0006] calling an image classification model to perform feature recognition processing on the first sample image to obtain a first feature map set, the first feature map set comprising a first intermediate feature map and a first target feature map, and calling the image classification model to perform feature recognition processing on the second sample image to obtain a second feature map set, the second feature map set comprising a second intermediate feature map and a second target feature map;

[0007] determining a value of a point loss function based on pixel point information of the first feature map set and pixel point information of the second feature map set;

[0008] determine a value of a contrast loss function based on a feature difference between the first target feature map and the second target feature map, and perform image classification processing on the first sample image and the second sample image based on the first target feature map and the second target feature map respectively, and determine a value of a classification loss function according to a result of the image classification processing and the first category label and the second category label;

[0009] determine a value of a training loss function corresponding to the image classification model according to the value of the point loss function, the value of the contrast loss function, and the value of the classification loss function, and train the image classification model in a direction of reducing the value of the training loss function.

[0010] In one aspect, an embodiment of the present application provides an image processing apparatus, comprising:

[0011] an acquisition unit configured to acquire a training sample pair, the training sample pair comprising a first sample image, a second sample image, a first category label corresponding to the first sample image, and a second category label corresponding to the second sample image;

[0012] a processing unit configured to perform feature recognition processing on the first sample image by invoking an image classification model to obtain a first feature map set, the first feature map set comprising a first intermediate feature map and a first target feature map, and perform feature recognition processing on the second sample image by invoking the image classification model to obtain a second feature map set, the second feature map set comprising a second intermediate feature map and a second target feature map;

[0013] The processing unit is further configured to determine a value of a point loss function based on pixel point information of the first feature map set and pixel point information of the second feature map set.

[0014] The processing unit is further configured to determine a value of a contrast loss function based on a feature difference between the first target feature map and the second target feature map, and perform image classification processing on the first sample image and the second sample image based on the first target feature map and the second target feature map respectively, and determine a value of a classification loss function according to a result of the image classification processing and the first category label and the second category label.

[0015] The processing unit is further configured to determine a value of a training loss function corresponding to the image classification model according to the value of the point loss function, the value of the contrast loss function, and the value of the classification loss function, and train the image classification model in a direction of reducing the value of the training loss function.

[0016] In one aspect, the embodiment of the present application provides an image processing device, characterized in that the image processing device comprises an input interface and an output interface, and further comprises:

[0017] a processor adapted to implement one or more instructions; and

[0018] a computer storage medium, the computer storage medium storing one or more instructions, the one or more instructions being adapted to be loaded by the processor and execute the image processing method.

[0019] In one aspect, the embodiment of the present application provides a computer storage medium, characterized in that the computer storage medium stores computer program instructions, and the computer program instructions are executed by a processor to execute the image processing method.

[0020] In one aspect, the embodiment of the present application provides a computer program product or a computer program, the computer program product comprising a computer program stored in a computer storage medium; the processor of the image processing device reads the computer program from the computer storage medium, and the processor executes the computer program, so that the image processing device executes the image processing method.

[0021] In the embodiments of the present application, the training sample pair including the first sample image, the second sample image, the first category label corresponding to the first sample image, and the second category label corresponding to the second sample image can be used to train the image classification model. Specifically, the value of the point loss function can be determined based on the pixel point information of the first feature map set and the pixel point information of the second feature map set. The first feature map set is obtained by calling the image classification model to perform feature recognition processing on the first sample image, and the second feature map set is obtained by calling the image classification model to perform feature recognition processing on the second sample image. The first feature map set includes a first intermediate feature map and a first target feature map, and the second feature map set includes a second intermediate feature map and a second target feature map. The value of the contrast loss function is determined based on the feature difference between the first target feature map and the second target feature map. The image classification processing is performed on the first sample image and the second sample image based on the first target feature map and the second target feature map respectively, and the value of the classification loss function is determined according to the results of the image classification processing and the first category label and the second category label. The value of the training loss function corresponding to the image classification model is determined according to the value of the point loss function, the value of the contrast loss function, and the value of the classification loss function, and the image classification model is trained in the direction of reducing the value of the training loss function. The local features of the pixel points can be learned through the point loss function, the feature difference between the first sample image and the second sample image can be learned through the contrast loss function, the difference between the classification processing result and the category label can be learned through the classification loss function, and the training accuracy of the image classification model can be improved through the design of the training loss function, thereby improving the classification accuracy of the image classification model, i.e. improving the image classification accuracy, especially improving the classification accuracy of the image with a small feature area. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0023] Figure 1 is a brief schematic diagram of an image processing scheme provided by an embodiment of the present application;

[0024] Figure 2 is a structural schematic diagram of an image classification model provided by an embodiment of the present application;

[0025] Figure 3 is a structural schematic diagram of a residual layer provided by an embodiment of the present application;

[0026] Figure 4is a schematic diagram of image processing by an image classification model provided by an embodiment of the present application;

[0027] Figure 5 is another schematic diagram of image processing by an image classification model provided by an embodiment of the present application;

[0028] Figure 6 is a flow schematic diagram of an image processing method provided by an embodiment of the present application;

[0029] Figure 7a is a schematic diagram of a second sample image provided by an embodiment of the present application;

[0030] Figure 7b is a schematic diagram of a first sample image provided by an embodiment of the present application;

[0031] Figure 8 is a schematic diagram of a first intermediate feature map provided by an embodiment of the present application;

[0032] Figure 9 is a schematic diagram of determining a first point loss value of a point loss function provided by an embodiment of the present application;

[0033] Figure 10 is a schematic diagram of determining a value of a training loss function provided by an embodiment of the present application;

[0034] Figure 11 is a schematic diagram of image classification on a to-be-processed image provided by an embodiment of the present application;

[0035] Figure 12 is a schematic diagram of determining a prediction value and a reference value corresponding to each pixel point in a feature map provided by an embodiment of the present application;

[0036] Figure 13 is another schematic diagram of determining a prediction value and a reference value corresponding to each pixel point in a feature map provided by an embodiment of the present application;

[0037] Figure 14 is a schematic diagram of a first region of a first intermediate feature map provided by an embodiment of the present application;

[0038] Figure 15 is a structural schematic diagram of an image processing apparatus provided by an embodiment of the present application;

[0039] Figure 16 is a structural schematic diagram of an image processing device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0040] With reference to the drawings and the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all the other embodiments obtained by those of ordinary skill in the art without creative efforts are within the scope of the present application.

[0041] Artificial Intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain optimal results. In other words, artificial intelligence is a comprehensive technology of computer science, which attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0042] Artificial intelligence technology is a comprehensive discipline, involving a wide range of fields, both hardware and software technologies. Artificial intelligence basic technologies generally include technologies such as sensors, special artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction system, mechatronics, etc. Artificial intelligence software technology mainly includes computer vision technology (CV), speech processing technology, natural language processing technology, and machine learning (ML) / deep learning (DL) and other major directions.

[0043] Among them, computer vision is a science that studies how to make machines "see". Further, it refers to using cameras and computers to replace human eyes to identify, track and measure targets, and further process graphics, so that computer processing becomes images more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision researches related theories and technologies, trying to establish artificial intelligence systems that can obtain information from images or multidimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (Optical Character Recognition, OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional (3-dimensional, 3D) technology, three-dimensional object reconstruction, virtual reality, augmented reality, simultaneous localization and mapping, etc. Among them, image processing technology can include image binarization, image compression and other technologies; image recognition technology can include image abstraction, similar image detection / de-duplication and other technologies; image semantic understanding technology can include image classification, image segmentation and other technologies; video semantic understanding technology can include video keyword labeling, video abstraction and other technologies. Machine learning is a multi-disciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It is a subject that studies how computers simulate or implement human learning behavior to acquire new knowledge or skills, and reorganize existing knowledge structure to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental approach to making computers intelligent. Its applications are widespread in various fields of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and rule-based learning.

[0044] Based on the above-mentioned machine learning techniques and the image classification techniques in the above-mentioned computer vision techniques, the embodiments of the present application provide an image processing scheme, which can train an image classification model by using a training sample pair including a first sample image and a second sample image. By designing a training loss function of the image classification model, the training accuracy of the image classification model can be improved, and further the classification accuracy of the image classification model can be improved, that is, the classification accuracy of the image can be improved. Among them, for the design of the training loss function, the training loss function can be designed to include a point loss function for learning local features of a pixel point, a contrast loss function for learning feature differences between the first sample image and the second sample image, and a classification loss function for learning the accuracy of the classification processing result obtained by the image classification model. For example, Figure 1As shown, a schematic diagram of an image processing scheme provided by an embodiment of the present application is shown. The first sample image and the second sample image can be processed by the image classification model to obtain a value of a training loss function composed of a value of a point loss function, a value of a contrast loss function and a value of a classification loss function, and the image classification model is trained in a direction of reducing the value of the training loss function. The first sample image and the second sample image share the model parameters of the image classification model, that is, the model parameters of the image classification model when processing the first sample image and the second sample image are consistent. The image classification model trained by the above image processing scheme can better recognize local features in the image, thereby improving the image classification accuracy. The image classification model trained above can be applied in multiple fields related to image recognition and classification such as autonomous driving and intelligent transportation.

[0045] In a specific implementation, the image processing scheme can be executed by an image processing device, which can be a terminal device or a server. The terminal device can include, but is not limited to, a computer, a smartphone, a tablet computer, a notebook computer, etc. The server can be a standalone physical server, a server cluster or a distributed system composed of multiple physical servers, a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and basic cloud computing services such as big data and artificial intelligence platforms.

[0046] The image classification model is a model for classifying images, i.e., the image classification model can be used to predict an image as a preset category among multiple preset categories to classify the image. The image processing device can set the preset categories according to specific training requirements, so that the image classification model can classify images of different preset categories. If the training requirement requires the image classification model to perform binary classification, the multiple preset categories can be set as a first preset category and a second preset category. In one application scenario, if the training requirement specifically requires the image classification model to perform binary classification on portrait images and non-portrait images, the first preset category can be a portrait category, and the second preset category can be a non-portrait category. In another application scenario, if the training requirement requires the image classification model to perform binary classification on sensitive images containing sensitive content and normal images not containing sensitive content, the first preset category can be set as a sensitive category, and the second preset category can be set as a normal category. The sensitive content can be the content concerned in the training requirement. For example, if the training requirement requires the image classification model to be used to determine whether a traffic accident occurs on a road, the image classification model is required to accurately classify images containing traffic accident content, and the sensitive content can be traffic accident-related content such as vehicle collision or vehicle overturning. For another example, if the training requirement requires the image classification model to be used to determine whether the image contains blood content, the image classification model is required to accurately classify images containing blood content, and the sensitive content can be blood content. For ease of description, the preset categories are described as the first preset category and the second preset category in the embodiments of the present application.

[0047] In one embodiment, the image classification model can be a convolutional neural network (CNN) model. For example, the image classification model can be a residual network (ResNet) model in the convolutional neural network model, such as a 50-layer residual network (ResNet50) model. The image classification model can also be a model improved based on the residual network model, such as a model improved based on the ResNet50 model. The image classification model can include M convolutional layers and N residual blocks, where M is greater than or equal to 1, and N is greater than or equal to 2. Further, the image classification model can also include a pooling layer, a fully connected layer, and a normalization layer, where the normalization layer can be a softmax activation function layer. Figure 2As shown in FIG. 1, a structure diagram of an image classification model provided by an embodiment of the present application is shown, taking processing of a first sample image by the image classification model as an example, M convolutional layers and N residual layers in the image classification model are used to perform feature recognition processing on the first sample image to obtain a first target feature map; further, the first target feature map can be subjected to feature analysis processing by a pooling layer in the image classification model to obtain a first feature vector; the first feature vector is subjected to classification prediction processing by a fully connected layer and a normalization layer in the image classification model to obtain a first category prediction result. The residual layer in the image classification model can be composed of two convolutional layers and an identity layer, as shown in FIG. 2, which is a structure diagram of a residual layer provided by an embodiment of the present application. If the input of the residual layer is x, the input x is processed by a first convolutional layer, the processed result is activated by an activation function rule, and the activated result is processed by a second convolutional layer to obtain a result F(x) of the second convolutional layer processing. The identity layer is used to directly output the input x, that is, the result of adding the result F(x) of the second convolutional layer processing and the input x is taken as the processing result of the residual layer. Further, the processing result of the residual layer can be activated by the activation function rule. Figure 3

[0048] Further, the convolutional layers in the image classification model are used for convolution processing, and the residual layers in the image classification model are used for feature extraction processing, as shown in FIG. 3, which is a structure diagram of an image classification model provided by an embodiment of the present application. As shown in FIG. 3, the image classification model is composed of M convolutional layers, K residual layers, N-K residual layers, a pooling layer, a fully connected layer and a normalization layer. The M convolutional layers are used to perform convolution processing on a first sample image to obtain a convolution feature map of the first sample image. The K residual layers are used to perform feature extraction processing on the convolution feature map of the first sample image to obtain a first intermediate feature map, K is greater than or equal to 1 and less than or equal to N. The N-K residual layers are used to perform feature extraction processing on the first intermediate feature map to obtain a first target feature map. The pooling layer is used to perform feature analysis processing on the first target feature map to obtain a first feature vector. The fully connected layer and the normalization layer are used to perform classification prediction processing on the first feature vector to obtain a first category prediction result. Figure 4 Figure 5 ​​Another image processing diagram provided by the embodiment of the present application is provided through an image classification model, which includes three convolutional layers and six residual layers. Taking processing of a first sample image by the image classification model as an example, if the width of the first sample image is w and the height is h, when the image classification model processes the first sample image, a first convolutional layer in the image classification model can be called to perform convolutional processing on the first sample image to obtain a first convolutional feature map of the first sample image, which is a 32-dimensional feature map with a width of w / 2 and a height of h / 2. A second convolutional layer in the image classification model is used to perform convolutional processing on the first convolutional feature map of the first sample image to obtain a second convolutional feature map of the first sample image, which is a 64-dimensional feature map with a width of w / 4 and a height of h / 4. A third convolutional layer in the image classification model is used to perform convolutional processing on the second convolutional feature map of the first sample image to obtain a third convolutional feature map of the first sample image, which is a 128-dimensional feature map with a width of w / 4 and a height of h / 4. Three residual layers in the image classification model are used to perform feature extraction processing on the third convolutional feature map of the first sample image to obtain a first intermediate feature map, which is a 128-dimensional feature map with a width of w / 4 and a height of h / 4. Another three residual layers in the image classification model are used to perform feature extraction processing on the first intermediate feature map to obtain a first target feature map, which is a 128-dimensional feature map with a width of w / 4 and a height of h / 4. A pooling layer in the image classification model is used to perform feature analysis processing on the first target feature map to obtain a first feature vector. A fully connected layer and a normalization layer in the image classification model are used to perform classification prediction processing on the first feature vector to obtain a first category prediction result.

[0049] Based on the above image processing scheme, the embodiment of the present application provides an image processing method. Referring to Figure 6 The flowchart of the image processing method provided by the embodiment of the present application is shown. Figure 6 The image processing method shown can be executed by an image processing device. Figure 6 The image processing method shown can include the following steps:

[0050] S601, obtaining a training sample pair.

[0051] The training sample pair includes a first sample image, a second sample image, a first category label corresponding to the first sample image, and a second category label corresponding to the second sample image. The first sample image and the second sample image have equal sizes. The first category label identifies the category of the first sample image, and the second category label identifies the category of the second sample image. If a first preset category is identified by 0 and a second preset category is identified by 1, the category of the first sample image is the first preset category, and the category of the second sample image is the second preset category, the first category label is 0, and the second category label is 1. Further, the first category label and the second category label can be represented in the form of a vector. If the correspondence between the vector elements of the vector and the preset categories is {the first preset category, the second preset category}, the category of the first sample image is the first preset category, and the category of the second sample image is the second preset category, the first category label can be {1, 0}, and the second category label can be {0, 1}.

[0052] In an embodiment, the first sample image can be an image of a first preset category or an image of a second preset category, and the second sample image can be an image of the first preset category or an image of the second preset category. The image processing device can obtain a training sample pair from a sample image set, and the sample image set includes one or more sample images of the first preset category and one or more sample images of the second preset category. Specifically, the image processing device can randomly select two sample images from the sample image set as the first sample image and the second sample image, respectively. When the first sample image and the second sample image are of the same category, the image classification model can learn the similarity between sample images of the same category, and when the first sample image and the second sample image are of different categories, the image classification model can learn the difference between sample images of different categories. For ease of description, the first sample image is of the first preset category and the second sample image is of the second preset category in the subsequent description. For example, if the image classification model is a model that can perform binary classification on sensitive images containing sensitive content and normal images not containing sensitive content, i.e., the first preset category is the sensitive category and the second preset category is the normal category, the first sample image can be a sensitive image or a normal image, and the second sample image can be a sensitive image or a normal image. Further, training the image classification model using sample images of different preset categories can enable the image classification model to classify images of different preset categories. For example, if the image classification model is trained using sensitive images of the sensitive category and normal images of the normal category, the image classification model can classify sensitive images and normal images. If the sensitive images used to train the image classification model are images containing traffic accident content and the normal images used to train the image classification model are images not containing traffic accident content, the image classification model can be applied to the intelligent transportation scenario to determine whether a traffic accident occurs on the road. If the sensitive images used to train the image classification model are images containing blood content and the normal images used to train the image classification model are images not containing blood content, the image classification model can be used to determine whether the image contains blood content.

[0053] Further, in order to improve the classification accuracy of the image classification model for images with small feature regions, sample images with small feature regions can be used to train the image classification model. Furthermore, in order to improve the classification accuracy of the image classification model, sample images with similar features can be used to train the image classification model. That is, the sample image set can include sample images of the first preset category that are easily misjudged as the second preset category, and sample images of the second preset category that are easily misjudged as the first preset category. For example, if the first preset category is a sensitive category and the second preset category is a normal category, if the sample images of the first preset category are identified as negative sample images and the sample images of the second preset category are identified as positive sample images, the sample images of the first preset category that are easily misjudged as the second preset category in the sample image set are sensitive images of highly suspected difficult sample images; the sample images of the second preset category that are easily misjudged as the first preset category in the sample image set are normal images of highly suspected difficult sample images, wherein the difficult sample images refer to normal images that are misjudged by the image classification model with a high score. For example, sample images of the normal category that are easily misjudged as the sensitive category (i.e., normal images of suspected sensitive images) can be normal images containing suspected blood content, such as normal images containing red liquid (e.g., red paint). As shown in Figure 7a FIG. 2 shows a schematic diagram of a second sample image provided by an embodiment of the present application, which is a normal image of a suspected sensitive image, wherein the red paint shown as 701 is suspected sensitive content; as shown in Figure 7b FIG. 3 shows a schematic diagram of a first sample image provided by an embodiment of the present application, wherein the blood content is shown as 702.

[0054] In S602, the image classification model is called to perform feature recognition processing on the first sample image to obtain a first feature map set, and the image classification model is called to perform feature recognition processing on the second sample image to obtain a second feature map set.

[0055] The first feature map set includes a first intermediate feature map and a first target feature map, and the second feature map set includes a second intermediate feature map and a second target feature map. The first intermediate feature map and the second intermediate feature map can be heat maps, which can be obtained by performing feature recognition processing by the image classification model and can represent the feature distribution in the first sample image and the second sample image. Referring to Figure 8 FIG. 4 shows a schematic diagram of a first intermediate feature map provided by an embodiment of the present application. If the first sample image is of the first preset category, the first intermediate feature map obtained by performing feature recognition processing by the image classification model can represent the region of the first preset category of the first sample image as shown by 801.

[0056] In an embodiment, the image classification model is a model for classifying images, and the image classification model can include M convolutional layers and N residual layers, where M is greater than or equal to 1, and N is greater than or equal to 2. In a specific implementation, the image processing device performs feature recognition processing on the first sample image by calling the image classification model to obtain a first feature map set, which can include: calling the M convolutional layers in the image classification model to perform convolutional processing on the first sample image to obtain a convolutional feature map of the first sample image; performing feature extraction processing on the convolutional feature map of the first sample image by K residual layers in the image classification model to obtain a first intermediate feature map, where K is greater than or equal to 1 and less than or equal to N; and performing feature extraction processing on the first intermediate feature map by N-K residual layers in the image classification model to obtain a first target feature map. The image processing device performs feature recognition processing on the second sample image by calling the image classification model to obtain a second feature map set, which can include: calling the M convolutional layers in the image classification model to perform convolutional processing on the second sample image to obtain a convolutional feature map of the second sample image; performing feature extraction processing on the convolutional feature map of the second sample image by K residual layers in the image classification model to obtain a second intermediate feature map, where K is greater than or equal to 1 and less than or equal to N; and performing feature extraction processing on the second intermediate feature map by N-K residual layers in the image classification model to obtain a second target feature map. The first sample image and the second sample image share model parameters of the image classification model, that is, the model parameters of the image classification model when processing the first sample image and the second sample image are consistent.

[0057] S603, determining a value of the point loss function based on pixel point information of the first feature map set and pixel point information of the second feature map set.

[0058] In one embodiment, the image processing device calls the image classification model to perform feature recognition processing on the first intermediate feature map and the first target feature map obtained from the first sample image, and calls the image classification model to perform feature recognition processing on the second intermediate feature map and the second target feature map obtained from the second sample image. The first intermediate feature map, the first target feature map, the second intermediate feature map, and the second target feature map are usually high-dimensional feature maps. Therefore, before determining the value of the point loss function based on the pixel point information of the first feature map set and the pixel point information of the second feature map set, the image processing device needs to first perform dimension reduction processing on the first intermediate feature map, the first target feature map, the second intermediate feature map, and the second target feature map to obtain one-dimensional first intermediate feature map, one-dimensional first target feature map, one-dimensional second intermediate feature map, and one-dimensional second target feature map. The subsequent processes of the image processing device for determining the value of the point loss function are all based on the processing of the one-dimensional first intermediate feature map, the one-dimensional first target feature map, the one-dimensional second intermediate feature map, and the one-dimensional second target feature map obtained after dimension reduction processing. In a specific implementation, the dimension reduction processing of the first intermediate feature map, the first target feature map, the second intermediate feature map, and the second target feature map can be implemented by performing maximum pooling processing on the first intermediate feature map, the first target feature map, the second intermediate feature map, and the second target feature map.

[0059] In one embodiment, the pixel point information of the first feature map set includes predicted values corresponding to L pixel points in the first intermediate feature map and reference values corresponding to L pixel points in the first target feature map, and the pixel point information of the second feature map set includes predicted values corresponding to L pixel points in the second intermediate feature map and reference values corresponding to L pixel points in the second target feature map, where L is the number of pixel points included in the first intermediate feature map, the first target feature map, the second intermediate feature map, and the second target feature map, and L is an integer greater than or equal to 1, that is, the sizes of the first intermediate feature map, the first target feature map, the second intermediate feature map, and the second target feature map are equal. The predicted value corresponding to one pixel point in the first intermediate feature map represents the probability of predicting the local feature of the pixel point as the class of the first sample image. If the first sample image is the first preset class, the predicted value corresponding to one pixel point in the first intermediate feature map represents the probability of predicting the local feature of the pixel point as the first preset class. The reference value corresponding to one pixel point in the first target feature map represents the probability of predicting the local feature of the pixel point as the class of the first sample image. If the first sample image is the first preset class, the reference value corresponding to one pixel point in the first target feature map represents the probability of predicting the local feature of the pixel point as the first preset class.

[0060] When the image processing device calls the image classification model to perform feature recognition processing on the first sample image to obtain the first intermediate feature map and the first target feature map, the convolution feature map of the first sample image is first processed by the K residual layers in the image classification model to obtain the first intermediate feature map; then the first intermediate feature map is processed by the N-K residual layers in the image classification model to obtain the first target feature map; therefore, the learning effect of the first target feature map obtained after the image classification model performs feature recognition processing on the class feature corresponding to the first preset category in the first sample image is better than that of the first intermediate feature map, that is, the feature extraction effect of the image classification model on the class feature corresponding to the first preset category through the N residual layers is better than that through only the K residual layers; therefore, the probabilities of the pixel points in the first target feature map can be used as reference values corresponding to the pixel points, the probabilities of the pixel points in the first intermediate feature map can be used as prediction values corresponding to the pixel points, and the first point loss value of the point loss function can be calculated based on the prediction values of the L pixel points in the first intermediate feature map and the reference values of the L pixel points in the first target feature map to train the image classification model, so that the first point loss value of the point loss function obtained based on the prediction values of the L pixel points in the first intermediate feature map and the reference values of the L pixel points in the first target feature map is small enough, that is, the prediction values of the L pixel points in the first intermediate feature map are close enough to the reference values of the L pixel points in the first target feature map, so as to train the image classification model to accurately extract the local features of the pixel points when extracting shallow features (that is, when the convolution feature map of the first sample image is processed by the first K residual layers in the image classification model), that is, to improve the learning effect of the image classification model on the local features.

[0061] Similarly, it can be known that the prediction value corresponding to one pixel point in the second intermediate feature map represents the probability of predicting the local feature possessed by the pixel point as the class of the second sample image, and if the second sample image is the second preset class, the prediction value corresponding to one pixel point in the second intermediate feature map represents the probability of predicting the local feature possessed by the pixel point as the second preset class. Similarly, the probability corresponding to the pixel point in the second target feature map can be taken as the reference value corresponding to the pixel point, the probability corresponding to the pixel point in the second intermediate feature map can be taken as the prediction value corresponding to the pixel point, and the second point loss value of the point loss function is calculated based on the prediction values corresponding to the L pixel points in the second intermediate feature map and the reference values corresponding to the L pixel points in the second target feature map. The image classification model is trained, and it is expected that the second point loss value of the point loss function obtained based on the prediction values corresponding to the L pixel points in the second intermediate feature map and the reference values corresponding to the L pixel points in the second target feature map is small enough, that is, the prediction values corresponding to the L pixel points in the second intermediate feature map and the reference values corresponding to the L pixel points in the second target feature map are close enough, so as to expect that the image classification model can accurately extract the local feature possessed by the pixel point when extracting the shallow feature (that is, when the convolution feature map of the second sample image is processed by the first K residual layers in the image classification model), that is, to improve the learning effect of the image classification model on the local feature.

[0062] Further, the first point loss value and the second point loss value are summed to obtain the value of the point loss function. That is, the image processing device determines the value of the point loss function based on the pixel point information of the first feature map set and the pixel point information of the second feature map set, which can include: determining the first point loss value of the point loss function based on the prediction values corresponding to the L pixel points in the first intermediate feature map and the reference values corresponding to the L pixel points in the first target feature map; determining the second point loss value of the point loss function based on the prediction values corresponding to the L pixel points in the second intermediate feature map and the reference values corresponding to the L pixel points in the second target feature map; and summing the first point loss value and the second point loss value to obtain the value of the point loss function.

[0063] In a specific implementation, the image processing device can determine the first point loss value of the point loss function based on the predicted values corresponding to the L pixel points in the first intermediate feature map and the reference values corresponding to the L pixel points in the first target feature map, which can include: determining the point loss value corresponding to each pixel point in the first intermediate feature map based on the predicted values corresponding to the L pixel points in the first intermediate feature map and the reference values corresponding to the L pixel points in the first target feature map; and performing summation operation on the point loss values corresponding to each pixel point in the first intermediate feature map to obtain the first point loss value of the point loss function.

[0064] Referring to Figure 9 An example of determining the first point loss value of the point loss function is provided for the embodiments of the present application. The image processing device can determine the point loss value corresponding to each pixel point in the first intermediate feature map based on the predicted values corresponding to the L pixel points in the first intermediate feature map and the reference values corresponding to the L pixel points in the first target feature map, which can include: for the i th pixel point in the first intermediate feature map, obtaining the predicted value corresponding to the i th pixel point, and obtaining the reference value corresponding to the j th pixel point in the first target feature map that matches the i th pixel point, where i is greater than or equal to 1 and less than or equal to L; and performing operation on the predicted value corresponding to the i th pixel point and the reference value corresponding to the j th pixel point using the pixel point loss operation rule to obtain the point loss value corresponding to the i th pixel point. The j th pixel point in the first target feature map that matches the i th pixel point is: the pixel point in the first target feature map that is located at the same position as the i th pixel point. The related process of the image processing device for determining the second point loss value of the point loss function based on the predicted values corresponding to the L pixel points in the second intermediate feature map and the reference values corresponding to the L pixel points in the second target feature map is the same as the above-mentioned related process of determining the first point loss value of the point loss function, and will not be described here.

[0065] Further, the pixel point loss operation rule is the operation rule of the point loss function, which can be used to calculate the first point loss value of the point loss function and the second point loss value of the point loss function. The point loss function can be given by the following formula (1.1):

[0066]

[0067] wherein i is an integer variable representing the i th pixel point; L is the number of pixel points in the first intermediate feature map, the first target feature map, the second intermediate feature map, and the second target feature map; and a and β are adjustment parameters that can be set according to specific training requirements, for example, a and β can both be set to 1. When the point loss function is used to determine the first point loss value of the point loss function, Y is the predicted value corresponding to the pixel point in the first intermediate feature map, is a reference value corresponding to a pixel point in the first target feature map, and Y is a prediction value corresponding to a pixel point in the second target feature map when the point loss function is used to determine the second point loss value of the point loss function, is a reference value corresponding to a pixel point in the second target feature map.

[0068] In an embodiment, taking the determination of the first point loss value of the point loss function as an example, the image processing device can perform an averaging operation on the results of the summation operation of the point loss values corresponding to each pixel point in the first intermediate feature map, to obtain the first point loss value of the point loss function. Then, the point loss function at this time can be given by the following formula (1.2):

[0069]

[0070] where i is an integer variable representing the ith pixel point, L is the number of pixel points in the first intermediate feature map, the first target feature map, the second intermediate feature map, and the second target feature map, and a and β are adjustment parameters that can be set according to specific training requirements, for example, a and β can both be set to 1. Y is a prediction value corresponding to a pixel point in the first intermediate feature map when the point loss function is used to determine the first point loss value of the point loss function, is a reference value corresponding to a pixel point in the first target feature map, and Y is a prediction value corresponding to a pixel point in the second target feature map when the point loss function is used to determine the second point loss value of the point loss function, is a reference value corresponding to a pixel point in the second target feature map.

[0071] In an embodiment, since the first point loss value of the point loss function is determined by the prediction values corresponding to each pixel point in the first intermediate feature map and the reference values corresponding to each pixel point in the first target feature map, and the second point loss value of the point loss function is determined by the prediction values corresponding to each pixel point in the second intermediate feature map and the reference values corresponding to each pixel point in the second target feature map, the distribution of the prediction values and the reference values in each feature map directly affects the size of the first point loss value and the second point loss value. The determination process of the prediction values and the reference values in the feature map is described in detail in subsequent embodiments.

[0072] S604, determining the value of the contrast loss function based on the feature difference between the first target feature map and the second target feature map, and performing image classification processing on the first sample image and the second sample image based on the first target feature map and the second target feature map respectively, and determining the value of the classification loss function according to the results of the image classification processing and the first class label and the second class label.

[0073] In one embodiment, the contrast loss function is used to learn the feature difference between the first sample image and the second sample image, that is, the similarity between sample images of the same category and the difference between sample images of different categories can be learned through the contrast loss function. In a specific implementation, the image processing device determines the value of the contrast loss function based on the feature difference between the first target feature map and the second target feature map, which can include: performing feature analysis processing on the first target feature map to obtain a first feature vector, and performing feature analysis processing on the second target feature map to obtain a second feature vector; using the contrast loss function operation rule to calculate the feature similarity between the first feature vector and the second feature vector to obtain the value of the contrast loss function.

[0074] wherein the first feature vector and the second feature vector are inputs of a fully connected layer in the image classification model, the first feature vector and the second feature vector are high-dimensional vectors, the vector dimension of the first feature vector and the second feature vector is related to the structure parameters of the specific image classification model, and if the image classification model is a ResNet50 model, the first feature vector and the second feature vector are 2048-dimensional vectors. Specifically, the first target feature map can be processed by a pooling layer in the image classification model to obtain the first feature vector, and the second target feature map can be processed by the pooling layer to obtain the second feature vector.

[0075] wherein the contrast loss function operation rule is the operation rule of the contrast loss function, which can be used to calculate the feature similarity between the first feature vector and the second feature vector, wherein the feature similarity between the first feature vector and the second feature vector can be represented by the norm between the first feature vector and the second feature vector, and the contrast loss function can be given by the following formula (2):

[0076] L con (X1,X2)=‖f(X1)-f(X2)‖ (2)

[0077] wherein X1 is the first feature vector and X2 is the second feature vector.

[0078] In an embodiment, the classification loss function is used to learn the accuracy of the results of the classification processing obtained by the image classification model; the image processing device performs image classification processing on the first sample image and the second sample image based on the first target feature map and the second target feature map respectively, and determines the value of the classification loss function according to the results of the image classification processing and the first category label and the second category label, which can include: performing feature analysis processing on the first target feature map to obtain a first feature vector, and performing feature analysis processing on the second target feature map to obtain a second feature vector; performing classification prediction processing on the first feature vector to obtain a first category prediction result, and performing classification prediction processing on the second feature vector to obtain a second category prediction result; determining the value of the classification loss function based on the difference between the first category prediction result and the first category label, and the difference between the second category prediction result and the second category label.

[0079] The first feature vector and the second feature vector are inputs of a fully connected layer in the image classification model, specifically, the first feature vector can be obtained by performing feature analysis processing on the first target feature map through a pooling layer in the image classification model, and the second feature vector can be obtained by performing feature analysis processing on the second target feature map through the pooling layer in the image classification model. Further, the first category prediction result can be obtained by performing classification prediction processing on the first feature vector through a fully connected layer and a normalization layer in the image classification model; and the second category prediction result can be obtained by performing classification prediction processing on the second feature vector through the fully connected layer and the normalization layer in the image classification model.

[0080] Further, the image processing device determines the value of the classification loss function based on the difference between the first category prediction result and the first category label, and the difference between the second category prediction result and the second category label, which can include: determining a first classification loss value of the classification loss function based on the difference between the first category prediction result and the first category label; determining a second classification loss value of the classification loss function based on the difference between the second category prediction result and the second category label; and performing summation operation on the first classification loss value and the second classification loss value to obtain the value of the classification loss function. Optionally, the classification loss function can be a cross-entropy loss function.

[0081] S605, determining the value of a training loss function corresponding to the image classification model according to the value of the point loss function, the value of the contrast loss function and the value of the classification loss function, and training the image classification model in a direction of reducing the value of the training loss function.

[0082] Wherein, the value of the point loss function includes a first point loss value of the point loss function determined based on the first intermediate feature map and the first target feature map, and a second point loss value of the point loss function determined based on the second intermediate feature map and the second target feature map; the value of the classification loss function includes a first classification loss value of the classification loss function determined based on the difference between the first category prediction result and the first category label, and a second classification loss value of the classification loss function determined based on the difference between the second category prediction result and the second category label; if the first point loss value is represented by L p1 , the second point loss value is represented by L p2 , the value of the contrast loss function is represented by L con , the first classification loss value is represented by L cls1 , and the second classification loss value is represented by L cls2 , then the value of the training loss function can be given by formula (3):

[0083] L loss = L p1 + L p2 + L cls1 + L cls2 + L con (3)

[0084] Based on the above description, the embodiment of the present application provides a schematic diagram for determining the value of the training loss function, as shown in Figure 10As shown, the image processing device calls the image classification model to perform feature recognition processing on the first sample image to obtain a first intermediate feature map and a first target feature map, and calls the image classification model to perform feature recognition processing on the second sample image to obtain a second intermediate feature map and a second target feature map; performs feature analysis processing on the first target feature map to obtain a first feature vector, and performs feature analysis processing on the second target feature map to obtain a second feature vector; performs classification prediction processing on the first feature vector to obtain a first category prediction result, and performs classification prediction processing on the second feature vector to obtain a second category prediction result. The image processing device determines a first point loss value of the point loss function based on the first intermediate feature map and the first target feature map, determines a second point loss value of the point loss function based on the second intermediate feature map and the second target feature map, determines a value of the contrast loss function based on the first target feature map and the second target feature map, determines a first classification loss value of the classification loss function based on the first category prediction result and a first category label corresponding to the first sample image, determines a second classification loss value of the classification loss function based on the second category prediction result and a second category label corresponding to the second sample image; and determines a value of the point loss function based on the first point loss value and the second point loss value, determines a value of the classification loss function based on the first classification loss value and the second classification loss value; and determines a value of the training loss function based on the value of the point loss function, the value of the contrast loss function, and the value of the classification loss function.

[0085] In one implementation, if the image classification model is trained based on H training sample pairs, the contrast loss function corresponding to the H training sample pairs can be given by formula (4):

[0086]

[0087] wherein i is an integer variable representing the i-th training sample pair, wherein is the first feature vector in the i-th training sample pair, is the second feature vector in the i-th training sample pair.

[0088] Further, the value of the training loss function corresponding to the H training sample pairs can be given by formula (5):

[0089]

[0090] wherein i is an integer variable representing the i-th training sample pair; L p1i represents the first point loss value corresponding to the i-th training sample pair; L p2i represents the second point loss value corresponding to the i-th training sample pair; L cls1i represents the first classification loss value corresponding to the i-th training sample pair; L cls2idenotes the second classification loss value corresponding to the i-th training sample pair; L conH denotes the contrast loss function value corresponding to the H training sample pairs.

[0091] In an embodiment, when the image classification model is trained, the image classification model can be trained based on the training sample pair including two sample images, and the value of the point loss function, the value of the contrast loss function, and the value of the classification loss function determined by the training sample pair; in order to improve the test efficiency of the image classification model, when the image classification model is tested, only one test image and the value of the classification loss function determined by the test image can be used to test the image classification model at a time, wherein the test image is an image different from the sample image used when the image classification model is trained. If the value of the point loss function is implemented by the point loss function determination module, the value of the contrast loss function is implemented by the contrast loss function determination module, and the value of the classification loss function is implemented by the classification loss function determination module, when the image classification model is tested, the point loss function determination module and the contrast loss function determination module can be disabled.

[0092] Further, after the image processing device completes training of the image classification model, a trained image classification model can be obtained; the trained image classification model can include M convolutional layers and N residual layers, and image classification processing can be implemented based on the trained image classification model. In a specific implementation, the image processing device can obtain a to-be-processed image, and call the M convolutional layers in the trained image classification model to perform convolutional processing on the to-be-processed image to obtain a convolutional feature map of the to-be-processed image; the K residual layers in the trained image classification model are used to perform feature extraction processing on the convolutional feature map of the to-be-processed image to obtain an intermediate feature map of the to-be-processed image; the N-K residual layers in the trained image classification model are used to perform feature extraction processing on the intermediate feature map of the to-be-processed image to obtain a target feature map of the to-be-processed image; the target feature map of the to-be-processed image is subjected to feature analysis processing to obtain a feature vector of the to-be-processed image; and the feature vector of the to-be-processed image is subjected to classification prediction processing to obtain a predicted class of the to-be-processed image. As shown in Figure 11 Further, after the image processing device completes training of the image classification model, a trained image classification model can be obtained; the trained image classification model can include M convolutional layers and N residual layers, and image classification processing can be implemented based on the trained image classification model. In a specific implementation, the image processing device can obtain a to-be-processed image, and call the M convolutional layers in the trained image classification model to perform convolutional processing on the to-be-processed image to obtain a convolutional feature map of the to-be-processed image; the K residual layers in the trained image classification model are used to perform feature extraction processing on the convolutional feature map of the to-be-processed image to obtain an intermediate feature map of the to-be-processed image; the N-K residual layers in the trained image classification model are used to perform feature extraction processing on the intermediate feature map of the to-be-processed image to obtain a target feature map of the to-be-processed image; the target feature map of the to-be-processed image is subjected to feature analysis processing to obtain a feature vector of the to-be-processed image; and the feature vector of the to-be-processed image is subjected to classification prediction processing to obtain a predicted class of the to-be-processed image. As shown in

[0093] In the embodiments of the present application, the training sample pair including the first sample image, the second sample image, the first category label corresponding to the first sample image, and the second category label corresponding to the second sample image can be used to train the image classification model. Specifically, the value of the point loss function can be determined based on the pixel point information of the first feature map set and the pixel point information of the second feature map set. The first feature map set is obtained by calling the image classification model to perform feature recognition processing on the first sample image, and the second feature map set is obtained by calling the image classification model to perform feature recognition processing on the second sample image. The first feature map set includes a first intermediate feature map and a first target feature map, and the second feature map set includes a second intermediate feature map and a second target feature map. The value of the contrast loss function is determined based on the feature difference between the first target feature map and the second target feature map. The image classification processing is performed on the first sample image and the second sample image based on the first target feature map and the second target feature map, respectively. The value of the classification loss function is determined according to the results of the image classification processing and the first category label and the second category label. The value of the training loss function corresponding to the image classification model is determined according to the value of the point loss function, the value of the contrast loss function, and the value of the classification loss function. The image classification model is trained in the direction of reducing the value of the training loss function. The local features of the pixel points can be learned through the point loss function, the feature difference between the first sample image and the second sample image can be learned through the contrast loss function, the difference between the results of the classification processing and the category labels can be learned through the classification loss function, and the training accuracy of the image classification model can be improved through the design of the training loss function. Thus, the classification accuracy of the image classification model is improved, and the classification accuracy of the image is improved, especially the classification accuracy of the image with a small feature region.

[0094] Based on the above image processing method, the determination of the predicted value corresponding to each pixel point in the first intermediate feature map and the determination of the reference value corresponding to each pixel point in the first target feature map directly affect the size of the first point loss value of the point loss function. The determination of the predicted value corresponding to each pixel point in the second intermediate feature map and the determination of the reference value corresponding to each pixel point in the second target feature map directly affect the size of the second point loss value of the point loss function. Based on this, the determination process of the predicted value and the reference value corresponding to each pixel point in the feature map is introduced in the embodiments of the present application. The determination process of the predicted value corresponding to each pixel point in the first intermediate feature map is the same as the determination process of the predicted value corresponding to each pixel point in the second intermediate feature map, and the determination process of the reference value corresponding to each pixel point in the first target feature map is the same as the determination process of the reference value corresponding to each pixel point in the second target feature map. Therefore, the determination of the predicted value corresponding to each pixel point in the first intermediate feature map and the determination of the reference value corresponding to each pixel point in the first target feature map are taken as examples for description. Referring to Figure 12A determination flow diagram of a prediction value and a reference value corresponding to each pixel point in a feature map provided for an embodiment of the present application is shown. Figure 12 The determination flow of the prediction value and the reference value corresponding to each pixel point in the feature map shown can be executed by an image processing device, and can specifically include the following steps:

[0095] S1201, determine a target pixel point of the first intermediate feature map and a prediction value corresponding to the target pixel point of the first intermediate feature map from L pixel points of the first intermediate feature map.

[0096] In a specific implementation, the image processing device determines a target pixel point of any feature map from L pixel points of the any feature map, and the any feature map includes the first intermediate feature map or the first target feature map, which can include: obtaining a plurality of connected domains in the any feature map; determining a connected domain that meets a preset condition in the plurality of connected domains in the any feature map as a target connected domain of the any feature map; and determining a center point of the target connected domain of the any feature map as the target pixel point of the any feature map. When the image processing device determines the target pixel point of the first intermediate feature map from the L pixel points of the first intermediate feature map, it can include: obtaining a plurality of connected domains in the first intermediate feature map; determining a connected domain that meets a preset condition in the plurality of connected domains in the first intermediate feature map as a target connected domain of the first intermediate feature map; and determining a center point of the target connected domain of the first intermediate feature map as the target pixel point of the first intermediate feature map.

[0097] The connected domain refers to an image region composed of pixel points with the same pixel value and adjacent positions in an image; and the connected domain of the first intermediate feature map refers to an image region composed of pixel points with the same pixel value and adjacent positions in the first intermediate feature map. The image processing device determines a connected domain that meets a preset condition in the plurality of connected domains in the first intermediate feature map as a target connected domain of the first intermediate feature map, in order to determine a region in the first intermediate feature map in which pixel points with a class feature corresponding to a class of the first sample image are concentrated; if the class of the first sample image is a first preset class, the target connected domain of the first intermediate feature map is a region in the first intermediate feature map in which pixel points with a class feature corresponding to the first preset class are concentrated. The preset condition is a condition for determining the target connected domain of the first intermediate feature map, for example, the preset condition can indicate that a connected domain composed of a preset pixel value in the plurality of connected domains in the first intermediate feature map is determined as the target connected domain of the first intermediate feature map, for example, the preset pixel value can be a pixel value corresponding to red; for another example, the preset condition can indicate that a connected domain with the largest area in the plurality of connected domains in the first intermediate feature map is determined as the target connected domain of the first intermediate feature map.

[0098] Further, the prediction value corresponding to the target pixel point of the first intermediate feature map can be determined as 1. The prediction value corresponding to the target pixel point of the first intermediate feature map is determined as 1 to ensure that the image classification model sufficiently learns the local features of the target pixel point.

[0099] S1202, based on the prediction value corresponding to the target pixel point of the first intermediate feature map and the prediction value distribution rule, determining the prediction value corresponding to each pixel point in the first intermediate feature map.

[0100] The prediction value distribution rule indicates that the farther the pixel point is from the target pixel point of the first intermediate feature map, the smaller the corresponding prediction value is.

[0101] In one embodiment, the prediction value distribution rule can be a distribution rule based on a Gaussian kernel function. Specifically, when the image processing device determines the prediction value corresponding to each pixel point in the first intermediate feature map based on the prediction value corresponding to the target pixel point of the first intermediate feature map and the prediction value distribution rule, the image processing device can input the position information corresponding to each pixel point in the first intermediate feature map and the position information of the target pixel point of the first intermediate feature map into the Gaussian kernel function for operation processing, and determine the obtained Gaussian kernel function value corresponding to each pixel point in the first intermediate feature map as the prediction value corresponding to each pixel point in the first intermediate feature map. The position information corresponding to each pixel point in the first intermediate feature map is the position coordinates of each pixel point in the first intermediate feature map. Specifically, a rectangular coordinate system can be established in the first intermediate feature map to determine the position coordinates of each pixel point. If the position coordinates of the target pixel point of the first intermediate feature map are (x p1 ,y p1 ), the Gaussian kernel function can be given by the following formula (6):

[0102]

[0103] where Y is the value of the Gaussian kernel function, i.e., the prediction value corresponding to the pixel point in the first intermediate feature map; σ p1 is a standard deviation related to the area of the target connected domain of the first intermediate feature map. If the area of the target connected domain of the first intermediate feature map is s1, then σ p1 is a standard deviation related to .

[0104] S1203, determining the target pixel point of the first target feature map and the reference value corresponding to the target pixel point of the first target feature map from the L pixel points of the first target feature map.

[0105] The process of determining the target pixel point of the first target feature map from the L pixel points of the first target feature map and the reference value corresponding to the target pixel point of the first target feature map is similar to the process of determining the target pixel point of the first intermediate feature map from the L pixel points of the first intermediate feature map and the predicted value corresponding to the target pixel point of the first intermediate feature map, and will not be described here.

[0106] In S1204, the reference value corresponding to each pixel point in the first target feature map is determined based on the reference value corresponding to the target pixel point of the first target feature map and the reference value distribution rule.

[0107] The reference value distribution rule indicates that the farther the pixel point is from the target pixel point of the first target feature map, the smaller the reference value corresponding to the pixel point is.

[0108] In an embodiment, the reference value distribution rule can be a distribution rule based on a Gaussian kernel function. Specifically, when the image processing device determines the reference value corresponding to each pixel point in the first target feature map based on the reference value corresponding to the target pixel point of the first target feature map and the reference value distribution rule, the image processing device can input the position information corresponding to each pixel point in the first target feature map and the position information of the target pixel point of the first target feature map into the Gaussian kernel function for operation processing, and determine the Gaussian kernel function value corresponding to each pixel point in the first target feature map as the reference value corresponding to each pixel point in the first target feature map. The position information corresponding to each pixel point in the first target feature map is the position coordinates of each pixel point in the first target feature map. Specifically, a rectangular coordinate system can be established in the first target feature map to determine the position coordinates of each pixel point. If the position coordinates of the target pixel point of the first target feature map are (x p2 ,y p2 ), the Gaussian kernel function can be given by the following formula (7):

[0109]

[0110] wherein, is the value of the Gaussian kernel function, i.e., the reference value corresponding to the pixel point in the first target feature map; σ p2 is a standard deviation related to the area of the target connected domain of the first target feature map. If the area of the target connected domain of the first target feature map is s2, then σ p2 is a standard deviation related to .

[0111] In the embodiment of the present application, the target pixel point of the first intermediate feature map and the predicted value corresponding to the target pixel point of the first intermediate feature map can be determined from the L pixel points of the first intermediate feature map. Then, the predicted value corresponding to each pixel point in the first intermediate feature map is determined based on the predicted value corresponding to the target pixel point of the first intermediate feature map and the predicted value distribution rule. The target pixel point of the first target feature map and the reference value corresponding to the target pixel point of the first target feature map are determined from the L pixel points of the first target feature map. The reference value corresponding to each pixel point in the first target feature map is determined based on the reference value corresponding to the target pixel point of the first target feature map and the reference value distribution rule. The predicted value distribution rule indicates that the farther the pixel point is from the target pixel point of the first intermediate feature map in the first intermediate feature map, the smaller the corresponding predicted value is. The reference value distribution rule indicates that the farther the pixel point is from the target pixel point of the first target feature map in the first target feature map, the smaller the corresponding reference value is. That is, by setting the predicted value and the reference value of the target pixel point to the maximum value, it can be ensured that the image classification model can accurately learn the local features of the target pixel point.

[0112] Based on the above image processing method, the embodiment of the present application provides another determination process of the predicted value and the reference value corresponding to each pixel point in the feature map. Referring to Figure 13 , another determination process of the predicted value and the reference value corresponding to each pixel point in the feature map provided by the embodiment of the present application is shown in the figure. Since the determination process of the predicted value corresponding to each pixel point in the first intermediate feature map is the same as the determination process of the predicted value corresponding to each pixel point in the second intermediate feature map, and the determination process of the reference value corresponding to each pixel point in the first target feature map is the same as the determination process of the reference value corresponding to each pixel point in the second target feature map, based on this, Figure 13 The determination process of the predicted value and the reference value corresponding to each pixel point in the feature map shown in the figure is described by taking the determination of the predicted value corresponding to each pixel point in the first intermediate feature map and the determination of the reference value corresponding to each pixel point in the first target feature map as an example. Figure 13 The determination process of the predicted value and the reference value corresponding to each pixel point in the feature map shown in the figure can be executed by an image processing device, and specifically can include the following steps:

[0113] S1301, the target pixel point of the first intermediate feature map and the predicted value corresponding to the target pixel point of the first intermediate feature map are determined from the L pixel points of the first intermediate feature map. And the first region and the second region of the first intermediate feature map are determined based on the target pixel point of the first intermediate feature map.

[0114] Among them, the first region of the first intermediate feature map includes the target pixel point of the first intermediate feature map, and the second region of the first intermediate feature map is the region other than the first region in the first intermediate feature map.

[0115] In an embodiment, when the image processing device determines the target pixel point of the first intermediate feature map from the L pixel points of the first intermediate feature map, it can include: obtaining a plurality of connected domains in the first intermediate feature map; determining a connected domain that meets a preset condition in the plurality of connected domains in the first intermediate feature map as the target connected domain of the first intermediate feature map; and determining the center point of the target connected domain of the first intermediate feature map as the target pixel point of the first intermediate feature map. Further, the prediction value corresponding to the target pixel point of the first intermediate feature map can be determined as 1, and the prediction value corresponding to the target pixel point of the first intermediate feature map is determined as 1 to ensure that the image classification model fully learns the local features of the target pixel point.

[0116] In an embodiment, the image processing device can determine the first region of the first intermediate feature map based on the target pixel point of the first intermediate feature map and the area of the target connected domain of the first intermediate feature map. In a specific implementation, the image processing device can construct a point supervision label based on the position information of the target pixel point of the first intermediate feature map and the area of the target connected domain of the first intermediate feature map, and determine the first region of the first intermediate feature map based on the point supervision label. The point supervision label includes the position information of the target pixel point and the square root of the area of the target connected domain of the first intermediate feature map, for example: if the position information of the target pixel point of the first intermediate feature map is the position coordinates p(x p1 ,y p1 ), and the area of the target connected domain of the first intermediate feature map is s1, then the point supervision label can be Further, when the image processing device determines the first region of the first intermediate feature map based on the point supervision label, it can determine a rectangular region as the first region based on the point supervision label, the center point of the rectangular region is the target pixel point of the first intermediate feature map, and the width and height of the rectangular region are the square root of the area of the target connected domain of the first intermediate feature map. As shown in Figure 14 , it is a schematic diagram of the first region of the first intermediate feature map provided by an embodiment of the present application, wherein the target pixel point of the first intermediate feature map is marked as 1401, and the first region is a rectangular region marked as 1402.

[0117] S1302, determining the prediction value corresponding to each pixel point in the first region based on the prediction value corresponding to the target pixel point of the first intermediate feature map and the first prediction value distribution rule corresponding to the first region.

[0118] S1303, determining the prediction value corresponding to each pixel point in the second region based on the second prediction value distribution rule corresponding to the second region.

[0119] In an embodiment, in steps S1302-S1303, the first prediction value distribution rule can indicate that the prediction value corresponding to each pixel point in the first region except the target pixel point of the first intermediate feature map is determined as the first prediction value; and the second prediction value distribution rule can indicate that the prediction value corresponding to each pixel point in the second region is determined as the second prediction value, and the second prediction value is smaller than the first prediction value. For example, the first prediction value can be set as the prediction value corresponding to the target pixel point of the first intermediate feature map, i.e., the first prediction value is set as 1, and the second prediction value is set as 0, i.e., the prediction value corresponding to each pixel point in the first region is set as 1, and the prediction value corresponding to each pixel point in the second region is set as 0. The prediction value corresponding to each pixel point in the first region is set to be greater than the prediction value corresponding to each pixel point in the second region, so that the image classification model can sufficiently learn the local features included in the first region.

[0120] In an embodiment, the first prediction value distribution rule can indicate that the farther the pixel point is from the target pixel point of the first intermediate feature map in the first region, the smaller the corresponding prediction value is; and the second prediction value distribution rule indicates that the prediction value corresponding to each pixel point in the second region of the first intermediate feature map is smaller than the prediction value corresponding to each pixel point in the first region of the first intermediate feature map. Optionally, the prediction value distribution rule can be a distribution rule based on a Gaussian kernel function. Specifically, when determining the prediction value corresponding to each pixel point in the first region based on the prediction value corresponding to the target pixel point of the first intermediate feature map and the first prediction value distribution rule corresponding to the first region, the image processing device can input the position information corresponding to each pixel point in the first region and the position information of the target pixel point of the first intermediate feature map into the Gaussian kernel function for operation processing, and determine the obtained Gaussian kernel function value corresponding to each pixel point in the first region as the prediction value corresponding to each pixel point in the first region. Optionally, the prediction value corresponding to each pixel point in the second region can be set as 0.

[0121] S1304, determining the target pixel point of the first target feature map and the reference value corresponding to the target pixel point of the first target feature map from the L pixel points of the first target feature map, and determining the third region and the fourth region of the first target feature map based on the target pixel point of the first target feature map.

[0122] The third region of the first target feature map includes the target pixel point of the first target feature map, and the fourth region of the first target feature map is other region of the first target feature map except the third region.

[0123] S1305, determining the reference value corresponding to each pixel point in the third region based on the reference value corresponding to the target pixel point of the first target feature map and the first reference value distribution rule corresponding to the third region.

[0124] The first reference value distribution rule indicates that the farther a pixel point is from the target pixel point of the first target feature map, the smaller the reference value corresponding to the pixel point is.

[0125] In S1306, the reference value corresponding to each pixel point in the fourth region is determined based on the second reference value distribution rule corresponding to the fourth region.

[0126] The second reference value distribution rule indicates that the reference value corresponding to each pixel point in the fourth region of the first target feature map is smaller than the reference value corresponding to each pixel point in the third region of the first target feature map.

[0127] In steps S1304-S1306, the process of determining the reference value corresponding to each pixel point in the first target feature map is similar to the process of determining the predicted value corresponding to each pixel point in the first intermediate feature map in steps S1301-S1303, and thus is not described again.

[0128] In the embodiments of the present application, the first region and the second region can be determined according to the target pixel point of the first intermediate feature map (the first region can be determined by the determined point supervision label), and the predicted value corresponding to each pixel point in the first intermediate feature map can be determined according to the first predicted value distribution rule corresponding to the first region and the second predicted value distribution rule corresponding to the second region; and the third region and the fourth region can be determined according to the target pixel point of the first target feature map, and the reference value corresponding to each pixel point in the first target feature map can be determined according to the first reference value distribution rule corresponding to the third region and the second reference value distribution rule corresponding to the fourth region, and the predicted value corresponding to each pixel point in the first region is greater than the predicted value corresponding to each pixel point in the second region, and the reference value corresponding to each pixel point in the third region is greater than the reference value corresponding to each pixel point in the fourth region, which can ensure that the image classification model can accurately learn the local features of each pixel point in the first region and the third region.

[0129] Based on the above method embodiments, the embodiments of the present application provide an image processing device. Referring to Figure 15 The structure of the image processing device provided by the embodiments of the present application is shown in the figure, which can include an acquisition unit 1501 and a processing unit 1502. Figure 15 The image processing device shown in the figure can run the following units:

[0130] The acquisition unit 1501 is configured to acquire a training sample pair, the training sample pair including a first sample image, a second sample image, a first category label corresponding to the first sample image, and a second category label corresponding to the second sample image.

[0131] The processing unit 1502 is configured to call an image classification model to perform feature recognition processing on the first sample image to obtain a first feature map set, the first feature map set including a first intermediate feature map and a first target feature map, and call the image classification model to perform feature recognition processing on the second sample image to obtain a second feature map set, the second feature map set including a second intermediate feature map and a second target feature map.

[0132] The processing unit 1502 is further configured to determine a value of a point loss function based on pixel point information of the first feature map set and pixel point information of the second feature map set.

[0133] The processing unit 1502 is further configured to determine a value of a contrast loss function based on a feature difference between the first target feature map and the second target feature map, and perform image classification processing on the first sample image and the second sample image based on the first target feature map and the second target feature map respectively, and determine a value of a classification loss function according to a result of the image classification processing and the first category label and the second category label.

[0134] The processing unit 1502 is further configured to determine a value of a training loss function corresponding to the image classification model according to the value of the point loss function, the value of the contrast loss function, and the value of the classification loss function, and train the image classification model in a direction of reducing the value of the training loss function.

[0135] In an embodiment, the pixel point information of the first feature map set includes predicted values corresponding to L pixel points in the first intermediate feature map and reference values corresponding to L pixel points in the first target feature map, and the pixel point information of the second feature map set includes predicted values corresponding to L pixel points in the second intermediate feature map and reference values corresponding to L pixel points in the second target feature map, L being an integer greater than or equal to 1; when the processing unit 1502 determines the value of the point loss function based on the pixel point information of the first feature map set and the pixel point information of the second feature map set, the processing unit 1502 specifically performs the following operations:

[0136] determining a first point loss value of the point loss function based on the predicted values corresponding to the L pixel points in the first intermediate feature map and the reference values corresponding to the L pixel points in the first target feature map;

[0137] determining a second point loss value of the point loss function based on the predicted values corresponding to the L pixel points in the second intermediate feature map and the reference values corresponding to the L pixel points in the second target feature map;

[0138] performing summation operation on the first point loss value and the second point loss value to obtain the value of the point loss function.

[0139] In an embodiment, when determining the first point loss value of the point loss function based on the prediction value corresponding to each of the L pixel points in the first intermediate feature map and the reference value corresponding to each of the L pixel points in the first target feature map, the processing unit 1502 specifically performs the following operations:

[0140] determining the point loss value corresponding to each pixel point in the first intermediate feature map based on the prediction value corresponding to each of the L pixel points in the first intermediate feature map and the reference value corresponding to each of the L pixel points in the first target feature map;

[0141] performing summation operation on the point loss value corresponding to each pixel point in the first intermediate feature map to obtain the first point loss value of the point loss function.

[0142] In an embodiment, when determining the point loss value corresponding to each pixel point in the first intermediate feature map based on the prediction value corresponding to each of the L pixel points in the first intermediate feature map and the reference value corresponding to each of the L pixel points in the first target feature map, the processing unit 1502 specifically performs the following operations:

[0143] for the i-th pixel point in the first intermediate feature map, obtaining the prediction value corresponding to the i-th pixel point, and obtaining the reference value corresponding to the j-th pixel point in the first target feature map that matches the i-th pixel point, i is greater than or equal to 1 and less than or equal to L;

[0144] performing operation on the prediction value corresponding to the i-th pixel point and the reference value corresponding to the j-th pixel point by using the pixel point loss operation rule to obtain the point loss value corresponding to the i-th pixel point.

[0145] In an embodiment, the processing unit 1502 is further configured to:

[0146] determining a target pixel point of the first intermediate feature map and a prediction value corresponding to the target pixel point of the first intermediate feature map from the L pixel points of the first intermediate feature map;

[0147] determining the prediction value corresponding to each pixel point in the first intermediate feature map based on the prediction value corresponding to the target pixel point of the first intermediate feature map and a prediction value distribution rule; the prediction value distribution rule indicates that the farther the pixel point is from the target pixel point of the first intermediate feature map, the smaller the prediction value corresponding to the pixel point is;

[0148] determining a target pixel point of the first target feature map and a reference value corresponding to the target pixel point of the first target feature map from the L pixel points of the first target feature map;

[0149] determine, based on the reference value corresponding to the target pixel point of the first target feature map and a reference value distribution rule, a reference value corresponding to each pixel point in the first target feature map; the reference value distribution rule indicates that the farther a pixel point is from the target pixel point of the first target feature map, the smaller the reference value corresponding to the pixel point is.

[0150] In one embodiment, when the processing unit 1502 determines the target pixel point of any feature map from L pixel points of the feature map, the feature map includes the first intermediate feature map or the first target feature map, and the following operations are specifically performed:

[0151] obtain a plurality of connected domains in the any feature map;

[0152] determine, as the target connected domain of the any feature map, a connected domain in the plurality of connected domains in the any feature map that satisfies a preset condition;

[0153] determine, as the target pixel point of the any feature map, a center point of the target connected domain of the any feature map.

[0154] In one embodiment, the processing unit 1502 is further configured to:

[0155] determine, from the L pixel points of the first intermediate feature map, the target pixel point of the first intermediate feature map and a prediction value corresponding to the target pixel point of the first intermediate feature map; and determine, based on the target pixel point of the first intermediate feature map, a first region and a second region of the first intermediate feature map; the first region of the first intermediate feature map includes the target pixel point of the first intermediate feature map, and the second region of the first intermediate feature map is a region other than the first region in the first intermediate feature map;

[0156] determine, based on the prediction value corresponding to the target pixel point of the first intermediate feature map and a first prediction value distribution rule corresponding to the first region, a prediction value corresponding to each pixel point in the first region; the first prediction value distribution rule indicates that the farther a pixel point is from the target pixel point of the first intermediate feature map, the smaller the prediction value corresponding to the pixel point is.

[0157] determine, based on a second prediction value distribution rule corresponding to the second region, a prediction value corresponding to each pixel point in the second region; the second prediction value distribution rule indicates that the prediction value corresponding to each pixel point in the second region of the first intermediate feature map is smaller than the prediction value corresponding to each pixel point in the first region of the first intermediate feature map.

[0158] In one embodiment, the processing unit 1502 is further configured to:

[0159] determining a target pixel point of the first target feature map from the L pixel points of the first target feature map, and a reference value corresponding to the target pixel point of the first target feature map; and determining a third region and a fourth region of the first target feature map based on the target pixel point of the first target feature map, wherein the third region of the first target feature map includes the target pixel point of the first target feature map, and the fourth region of the first target feature map is a region other than the third region in the first target feature map;

[0160] determining a reference value corresponding to each pixel point in the third region based on the reference value corresponding to the target pixel point of the first target feature map and a first reference value distribution rule corresponding to the third region; the first reference value distribution rule indicates that the farther the pixel point is from the target pixel point of the first target feature map in the third region, the smaller the corresponding reference value is;

[0161] determining a reference value corresponding to each pixel point in the fourth region based on a second reference value distribution rule corresponding to the fourth region; the second reference value distribution rule indicates that the reference value corresponding to each pixel point in the fourth region of the first target feature map is smaller than the reference value corresponding to each pixel point in the third region of the first target feature map.

[0162] In one embodiment, when the processing unit 1502 determines the value of the contrast loss function based on the feature difference between the first target feature map and the second target feature map, the following operations are specifically performed:

[0163] performing feature analysis processing on the first target feature map to obtain a first feature vector, and performing feature analysis processing on the second target feature map to obtain a second feature vector;

[0164] using a contrast loss function operation rule to operate the feature similarity between the first feature vector and the second feature vector to obtain the value of the contrast loss function.

[0165] In one embodiment, when the processing unit 1502 respectively performs image classification processing on the first sample image and the second sample image based on the first target feature map and the second target feature map of the first sample image and the second sample image, and determines the value of the classification loss function according to the results of the image classification processing and the first class label and the second class label, the following operations are specifically performed:

[0166] performing feature analysis processing on the first target feature map to obtain a first feature vector, and performing feature analysis processing on the second target feature map to obtain a second feature vector;

[0167] performing classification prediction processing on the first feature vector to obtain a first category prediction result, and performing classification prediction processing on the second feature vector to obtain a second category prediction result;

[0168] determining the value of the classification loss function based on a difference between the first category prediction result and the first category label and a difference between the second category prediction result and the second category label.

[0169] In one embodiment, the image classification model comprises M convolution layers and N residual layers, M is greater than or equal to 1, and N is greater than or equal to 2.

[0170] When the processing unit 1502 invokes the image classification model to perform feature recognition processing on the first sample image to obtain a first feature map set, the following operations are performed:

[0171] The M convolution layers in the image classification model are invoked to perform convolution processing on the first sample image to obtain a convolution feature map of the first sample image.

[0172] The K residual layers in the image classification model are invoked to perform feature extraction processing on the convolution feature map of the first sample image to obtain the first intermediate feature map, K is greater than or equal to 1 and less than or equal to N.

[0173] The N-K residual layers in the image classification model are invoked to perform feature extraction processing on the first intermediate feature map to obtain the first target feature map.

[0174] In one embodiment, after the image classification model is trained;

[0175] The acquisition unit 1501 is further configured to acquire a to-be-processed image, and the processing unit 1502 is further configured to invoke the M convolution layers in the trained image classification model to perform convolution processing on the to-be-processed image to obtain a convolution feature map of the to-be-processed image.

[0176] The processing unit 1502 is further configured to perform feature extraction processing on the convolution feature map of the to-be-processed image by the K residual layers in the trained image classification model to obtain an intermediate feature map of the to-be-processed image.

[0177] The processing unit 1502 is further configured to perform feature extraction processing on the intermediate feature map of the to-be-processed image by the N-K residual layers in the trained image classification model to obtain a target feature map of the to-be-processed image.

[0178] The processing unit 1502 is further configured to perform feature analysis on the target feature map of the image to be processed to obtain a feature vector of the image to be processed;

[0179] The processing unit 1502 is further configured to perform classification prediction processing on the feature vector of the image to be processed to obtain a predicted category of the image to be processed.

[0180] According to one embodiment of the present application, Figure 6 、 Figure 12 as well as Figure 13 The steps involved in the image processing method shown can be Figure 15 The image processing apparatus shown in FIG. Figure 6 Step S601 shown can be performed by Figure 15 The acquisition unit 1501 in the image processing apparatus shown is executed, Figure 6 Steps S602 to S605 shown in FIG. Figure 15 The processing unit 1502 in the image processing apparatus shown in FIG. Figure 12 Steps S1201 to S1204 shown in FIG. Figure 15 The processing unit 1502 in the image processing apparatus shown in FIG. Figure 13 Steps S1301 to S1306 shown in FIG. Figure 15 The image processing apparatus shown is executed by the processing unit 1502.

[0181] According to another embodiment of the present application, Figure 15 The various units in the image processing device shown can be individually or all combined into one or several other units to constitute, or one (or some) of the units can be further divided into multiple smaller units in function to constitute, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In actual applications, the functions of one unit can also be implemented by multiple units, or the functions of multiple units can be implemented by one unit. In other embodiments of the present application, the image processing device based on logical function division can also include other units. In actual applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of multiple units.

[0182] According to another embodiment of the present application, the program can be executed by running on a general computing device such as a computer including a central processing unit (CPU), a random access memory (RAM), a read-only memory (ROM) and other processing elements and storage elements. Figure 6 、 Figure 12 or Figure 13A computer program (including program code) for each step involved in the corresponding method shown is used to construct an image processing device as shown in Figure 15 in the present application, and to implement the image processing method of the embodiments of the present application. The computer program can be recorded on, for example, a computer readable storage medium, and loaded into the above-mentioned computing device through the computer readable storage medium, and run therein.

[0183] In the embodiments of the present application, the training sample pair including the first sample image, the second sample image, the first class label corresponding to the first sample image and the second class label corresponding to the second sample image can be used to train the image classification model; specifically, the value of the point loss function can be determined based on the pixel point information of the first feature map set and the pixel point information of the second feature map set; wherein the first feature map set is obtained by calling the image classification model to perform feature recognition processing on the first sample image, and the second feature map set is obtained by calling the image classification model to perform feature recognition processing on the second sample image, the first feature map set includes the first intermediate feature map and the first target feature map, and the second feature map set includes the second intermediate feature map and the second target feature map; the value of the contrast loss function is determined based on the feature difference between the first target feature map and the second target feature map; and the first sample image and the second sample image are respectively processed by the image classification processing based on the first target feature map and the second target feature map, and the value of the classification loss function is determined according to the results of the image classification processing and the first class label and the second class label; the value of the training loss function corresponding to the image classification model is determined according to the value of the point loss function, the value of the contrast loss function and the value of the classification loss function, and the image classification model is trained in the direction of reducing the value of the training loss function. The local features of the pixel points can be learned through the point loss function, the feature difference between the first sample image and the second sample image can be learned through the contrast loss function, the difference between the results of the classification processing and the class labels can be learned through the classification loss function, and the training accuracy of the image classification model can be improved through the design of the training loss function, thereby improving the classification accuracy of the image classification model, i.e. improving the image classification accuracy, especially improving the classification accuracy of the image with small feature area.

[0184] Based on the above method embodiments and device embodiments, the present application further provides an image processing device. Referring to Figure 16 , a structural schematic diagram of an image processing device provided by the embodiments of the present application. Figure 16 The image processing device shown can at least include a processor 1601, an input interface 1602, an output interface 1603 and a computer storage medium 1604. Among them, the processor 1601, the input interface 1602, the output interface 1603 and the computer storage medium 1604 can be connected through bus or other ways.

[0185] The computer storage medium 1604 can be stored in the memory of the image processing device, and is used to store a computer program including program instructions. The processor 1601 is used to execute the program instructions stored in the computer storage medium 1604. The processor 1601 (also referred to as a CPU (Central Processing Unit, central processor)) is the computing core and control core of the image processing device, and is adapted to implement one or more instructions, and is specifically adapted to load and execute one or more instructions to implement the above-mentioned image processing method flow or corresponding functions.

[0186] The computer storage medium provided by the embodiment of the present application is a memory device in the image processing device, and is used to store programs and data. It can be understood that the computer storage medium herein can include a built-in storage medium in the terminal, and of course can also include an extended storage medium supported by the terminal. The computer storage medium provides a storage space, and the storage space stores an operating system of the terminal. In addition, one or more instructions adapted to be loaded and executed by the processor 1601 are also stored in the storage space, and the instructions can be one or more computer programs (including program codes). It should be noted that the computer storage medium herein can be a high-speed random access memory (RAM) memory, or a non-volatile memory such as at least one disk memory; and optionally can also be at least one computer storage medium located away from the aforementioned processor.

[0187] In one embodiment, one or more instructions stored in the computer storage medium can be loaded and executed by the processor 1601 and the input interface 1602 to implement the corresponding steps of the method in the above-mentioned image processing method embodiment Figure 6 、 Figure 12 and Figure 13 In a specific implementation, the one or more instructions in the computer storage medium are loaded and executed by the processor 1601 and the input interface 1602 as follows:

[0188] The input interface 1602 is used to obtain a training sample pair, and the training sample pair includes a first sample image, a second sample image, a first category label corresponding to the first sample image, and a second category label corresponding to the second sample image;

[0189] Processor 1601 is configured to call an image classification model to perform feature recognition processing on the first sample image to obtain a first feature atlas, where the first feature atlas includes a first intermediate feature map and a first target feature map; and call the image classification model to perform feature recognition processing on the second sample image to obtain a second feature atlas, where the second feature atlas includes a second intermediate feature map and a second target feature map;

[0190] The processor 1601 is further configured to determine a value of a point loss function based on the pixel information of the first feature atlas and the pixel information of the second feature atlas;

[0191] The processor 1601 is further configured to determine a value of a contrast loss function based on a feature difference between the first target feature map and the second target feature map, perform image classification processing on the first sample image and the second sample image based on the first target feature map and the second target feature map, respectively, and determine a value of a classification loss function based on a result of the image classification processing and the first category label and the second category label;

[0192] The processor 1601 is also used to determine the value of the training loss function corresponding to the image classification model based on the value of the point loss function, the value of the contrast loss function and the value of the classification loss function, and train the image classification model in the direction of reducing the value of the training loss function.

[0193] In one embodiment, the pixel information of the first feature atlas includes predicted values ​​corresponding to L pixels in the first intermediate feature map and reference values ​​corresponding to L pixels in the first target feature map; the pixel information of the second feature atlas includes predicted values ​​corresponding to L pixels in the second intermediate feature map and reference values ​​corresponding to L pixels in the second target feature map, where L is an integer greater than or equal to 1; when the processor 1601 determines the value of the point loss function based on the pixel information of the first feature atlas and the pixel information of the second feature atlas, it specifically performs the following operations:

[0194] Determining a first point loss value of the point loss function based on the predicted values ​​corresponding to the L pixels in the first intermediate feature map and the reference values ​​corresponding to the L pixels in the first target feature map;

[0195] Determining a second point loss value of the point loss function based on the predicted values ​​corresponding to the L pixels in the second intermediate feature map and the reference values ​​corresponding to the L pixels in the second target feature map;

[0196] A sum operation is performed on the first point loss value and the second point loss value to obtain a value of the point loss function.

[0197] In an embodiment, when determining the first point loss value of the point loss function based on the predicted value corresponding to each of the L pixel points in the first intermediate feature map and the reference value corresponding to each of the L pixel points in the first target feature map, the processor 1601 specifically performs the following operations:

[0198] determining the point loss value corresponding to each pixel point in the first intermediate feature map based on the predicted value corresponding to each of the L pixel points in the first intermediate feature map and the reference value corresponding to each of the L pixel points in the first target feature map;

[0199] performing summation operation on the point loss value corresponding to each pixel point in the first intermediate feature map to obtain the first point loss value of the point loss function.

[0200] In an embodiment, when determining the point loss value corresponding to each pixel point in the first intermediate feature map based on the predicted value corresponding to each of the L pixel points in the first intermediate feature map and the reference value corresponding to each of the L pixel points in the first target feature map, the processor 1601 specifically performs the following operations:

[0201] for the i-th pixel point in the first intermediate feature map, obtaining the predicted value corresponding to the i-th pixel point, and obtaining the reference value corresponding to the j-th pixel point in the first target feature map which matches the i-th pixel point, i is greater than or equal to 1 and less than or equal to L;

[0202] performing operation on the predicted value corresponding to the i-th pixel point and the reference value corresponding to the j-th pixel point by using the pixel point loss operation rule to obtain the point loss value corresponding to the i-th pixel point.

[0203] In an embodiment, the processor 1601 is further configured to:

[0204] determining the target pixel point of the first intermediate feature map and the predicted value corresponding to the target pixel point of the first intermediate feature map from the L pixel points of the first intermediate feature map;

[0205] determining the predicted value corresponding to each pixel point in the first intermediate feature map based on the predicted value corresponding to the target pixel point of the first intermediate feature map and the predicted value distribution rule; the predicted value distribution rule indicates that the farther the pixel point is from the target pixel point of the first intermediate feature map in the first intermediate feature map, the smaller the corresponding predicted value is;

[0206] determining the target pixel point of the first target feature map and the reference value corresponding to the target pixel point of the first target feature map from the L pixel points of the first target feature map;

[0207] determine, based on the reference value corresponding to the target pixel point of the first target feature map and a reference value distribution rule, a reference value corresponding to each pixel point in the first target feature map; the reference value distribution rule indicates that the farther a pixel point is from the target pixel point of the first target feature map, the smaller the reference value corresponding to the pixel point is.

[0208] In one embodiment, when the processor 1601 determines the target pixel point of any feature map from L pixel points of the feature map, the feature map includes the first intermediate feature map or the first target feature map, and the following operations are specifically performed:

[0209] obtain a plurality of connected domains in the any feature map;

[0210] determine, as a target connected domain of the any feature map, a connected domain in the plurality of connected domains in the any feature map that satisfies a preset condition;

[0211] determine, as the target pixel point of the any feature map, a center point of the target connected domain of the any feature map.

[0212] In one embodiment, the processor 1601 is further configured to:

[0213] determine, from the L pixel points of the first intermediate feature map, the target pixel point of the first intermediate feature map and a prediction value corresponding to the target pixel point of the first intermediate feature map; and determine, based on the target pixel point of the first intermediate feature map, a first region and a second region of the first intermediate feature map; the first region of the first intermediate feature map includes the target pixel point of the first intermediate feature map, and the second region of the first intermediate feature map is a region other than the first region in the first intermediate feature map;

[0214] determine, based on the prediction value corresponding to the target pixel point of the first intermediate feature map and a first prediction value distribution rule corresponding to the first region, a prediction value corresponding to each pixel point in the first region; the first prediction value distribution rule indicates that the farther a pixel point is from the target pixel point of the first intermediate feature map, the smaller the prediction value corresponding to the pixel point is.

[0215] determine, based on a second prediction value distribution rule corresponding to the second region, a prediction value corresponding to each pixel point in the second region; the second prediction value distribution rule indicates that the prediction value corresponding to each pixel point in the second region of the first intermediate feature map is smaller than the prediction value corresponding to each pixel point in the first region of the first intermediate feature map.

[0216] In one embodiment, the processor 1601 is further configured to:

[0217] determining a target pixel point of the first target feature map from the L pixel points of the first target feature map, and a reference value corresponding to the target pixel point of the first target feature map; and determining a third region and a fourth region of the first target feature map based on the target pixel point of the first target feature map, wherein the third region of the first target feature map includes the target pixel point of the first target feature map, and the fourth region of the first target feature map is a region other than the third region in the first target feature map;

[0218] determining a reference value corresponding to each pixel point in the third region based on the reference value corresponding to the target pixel point of the first target feature map and a first reference value distribution rule corresponding to the third region; the first reference value distribution rule indicates that the farther the pixel point is from the target pixel point of the first target feature map in the third region, the smaller the corresponding reference value is;

[0219] determining a reference value corresponding to each pixel point in the fourth region based on a second reference value distribution rule corresponding to the fourth region; the second reference value distribution rule indicates that the reference value corresponding to each pixel point in the fourth region of the first target feature map is smaller than the reference value corresponding to each pixel point in the third region of the first target feature map.

[0220] In one embodiment, when the processor 1601 determines the value of the contrast loss function based on the feature difference between the first target feature map and the second target feature map, the following operations are specifically performed:

[0221] performing feature analysis processing on the first target feature map to obtain a first feature vector, and performing feature analysis processing on the second target feature map to obtain a second feature vector;

[0222] using a contrast loss function operation rule to operate the feature similarity between the first feature vector and the second feature vector to obtain the value of the contrast loss function.

[0223] In one embodiment, when the processor 1601 respectively performs image classification processing on the first sample image and the second sample image based on the first target feature map and the second target feature map of the first sample image and the second sample image, and determines the value of the classification loss function according to the results of the image classification processing and the first class label and the second class label, the following operations are specifically performed:

[0224] performing feature analysis processing on the first target feature map to obtain a first feature vector, and performing feature analysis processing on the second target feature map to obtain a second feature vector;

[0225] performing classification prediction processing on the first feature vector to obtain a first category prediction result, and performing classification prediction processing on the second feature vector to obtain a second category prediction result;

[0226] determining the value of the classification loss function based on a difference between the first category prediction result and the first category label and a difference between the second category prediction result and the second category label.

[0227] In one embodiment, the image classification model comprises M convolution layers and N residual layers, M is greater than or equal to 1, and N is greater than or equal to 2.

[0228] When the processor 1601 invokes the image classification model to perform feature recognition processing on the first sample image to obtain a first feature map set, the following operations are specifically performed:

[0229] Invoke the M convolution layers in the image classification model to perform convolution processing on the first sample image to obtain a convolution feature map of the first sample image.

[0230] Perform feature extraction processing on the convolution feature map of the first sample image through the K residual layers in the image classification model to obtain the first intermediate feature map, K is greater than or equal to 1 and less than or equal to N.

[0231] Perform feature extraction processing on the first intermediate feature map through the N-K residual layers in the image classification model to obtain the first target feature map.

[0232] In one embodiment, after the image classification model is trained;

[0233] The input interface 1602 is further configured to acquire a to-be-processed image, and the processor 1601 is further configured to invoke the M convolution layers in the trained image classification model to perform convolution processing on the to-be-processed image to obtain a convolution feature map of the to-be-processed image.

[0234] The processor 1601 is further configured to perform feature extraction processing on the convolution feature map of the to-be-processed image through the K residual layers in the trained image classification model to obtain an intermediate feature map of the to-be-processed image.

[0235] The processor 1601 is further configured to perform feature extraction processing on the intermediate feature map of the to-be-processed image through the N-K residual layers in the trained image classification model to obtain a target feature map of the to-be-processed image.

[0236] The processor 1601 is further configured to perform feature analysis processing on the target feature map of the to-be-processed image to obtain a feature vector of the to-be-processed image.

[0237] The processor 1601 is further configured to perform classification prediction on the feature vector of the image to be processed, to obtain a predicted category of the image to be processed.

[0238] The embodiment of the present application provides a computer program product or a computer program. The computer program product comprises a computer program stored in a computer storage medium. A processor of an image processing device reads the computer program from the computer storage medium. The processor executes the computer program, so that the image processing device executes the method embodiment as shown in the above method embodiment. Figure 6 Figure 12 and Figure 13 The computer readable storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM) and the like.

[0239] The above merely provides the specific implementation of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of the changes or replacements within the technical range disclosed in the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.​

Claims

1. An image processing method, characterized by, The method comprises: obtaining a training sample pair, the training sample pair comprising a first sample image, a second sample image, a first category label corresponding to the first sample image, and a second category label corresponding to the second sample image; calling an image classification model to perform feature recognition processing on the first sample image to obtain a first feature map set, the first feature map set comprising a first intermediate feature map and a first target feature map, and calling the image classification model to perform feature recognition processing on the second sample image to obtain a second feature map set, the second feature map set comprising a second intermediate feature map and a second target feature map; the intermediate feature map is obtained from K residual layers, and the target feature map is obtained from N-K residual layers; pixel point information of the first feature map set comprises predicted values corresponding to L pixel points in the first intermediate feature map and reference values corresponding to L pixel points in the first target feature map; pixel point information of the second feature map set comprises predicted values corresponding to L pixel points in the second intermediate feature map and reference values corresponding to L pixel points in the second target feature map, L is an integer greater than or equal to 1, N is greater than or equal to 2, K is greater than or equal to 1 and less than or equal to N; determining a first point loss value of a point loss function based on the predicted values corresponding to the L pixel points in the first intermediate feature map and the reference values corresponding to the L pixel points in the first target feature map; determining a second point loss value of the point loss function based on the predicted values corresponding to the L pixel points in the second intermediate feature map and the reference values corresponding to the L pixel points in the second target feature map; performing summation operation on the first point loss value and the second point loss value to obtain a value of the point loss function; determining a value of a contrast loss function based on feature differences between the first target feature map and the second target feature map, and performing image classification processing on the first sample image and the second sample image based on the first target feature map and the second target feature map respectively, and determining a value of a classification loss function according to results of the image classification processing and the first category label and the second category label; determining a value of a training loss function corresponding to the image classification model according to the value of the point loss function, the value of the contrast loss function, and the value of the classification loss function, and training the image classification model in a direction of reducing the value of the training loss function.

2. The method of claim 1, wherein, The method further comprises: determining a point loss value corresponding to each pixel point in the first intermediate feature map based on the predicted values corresponding to the L pixel points in the first intermediate feature map and the reference values corresponding to the L pixel points in the first target feature map; performing summation operation on the point loss values corresponding to each pixel point in the first intermediate feature map to obtain the first point loss value of the point loss function.

3. The method of claim 2, wherein, The method further comprises: determining a target pixel point of the first intermediate feature map and a prediction value corresponding to the target pixel point of the first intermediate feature map from the L pixel points of the first intermediate feature map; determining the prediction value corresponding to each pixel point in the first intermediate feature map based on the prediction value corresponding to the target pixel point of the first intermediate feature map and a prediction value distribution rule; the prediction value distribution rule indicates that the farther the pixel point is from the target pixel point of the first intermediate feature map, the smaller the prediction value corresponding to the pixel point is; 4. The method of claim 2, wherein, determining a target pixel point of the first target feature map and a reference value corresponding to the target pixel point of the first target feature map from the L pixel points of the first target feature map; determining the reference value corresponding to each pixel point in the first target feature map based on the reference value corresponding to the target pixel point of the first target feature map and a reference value distribution rule; the reference value distribution rule indicates that the farther the pixel point is from the target pixel point of the first target feature map, the smaller the reference value corresponding to the pixel point is. The method further comprises: determining a target pixel point of the first intermediate feature map and a prediction value corresponding to the target pixel point of the first intermediate feature map from the L pixel points of the first intermediate feature map; and determining a first region and a second region of the first intermediate feature map based on the target pixel point of the first intermediate feature map; the first region of the first intermediate feature map includes the target pixel point of the first intermediate feature map, and the second region of the first intermediate feature map is other regions of the first intermediate feature map except the first region; determining a target pixel point of the first target feature map and a reference value corresponding to the target pixel point of the first target feature map from the L pixel points of the first target feature map; and determining a first region and a second region of the first target feature map based on the target pixel point of the first target feature map; the first region of the first target feature map includes the target pixel point of the first target feature map, and the second region of the first target feature map is other regions of the first target feature map except the first region.

5. The method of claim 4, wherein, The method further comprises: determining a target pixel point of the first intermediate feature map and a prediction value corresponding to the target pixel point of the first intermediate feature map from the L pixel points of the first intermediate feature map; and determining a first region and a second region of the first intermediate feature map based on the target pixel point of the first intermediate feature map; the first region of the first intermediate feature map includes the target pixel point of the first intermediate feature map, and the second region of the first intermediate feature map is other regions of the first intermediate feature map except the first region; determining a target pixel point of the first target feature map and a reference value corresponding to the target pixel point of the first target feature map from the L pixel points of the first target feature map; and determining a first region and a second region of the first target feature map based on the target pixel point of the first target feature map; the first region of the first target feature map includes the target pixel point of the first target feature map, and the second region of the first target feature map is other regions of the first target feature map except the first region. ​ 6. The method of claim 2, wherein, ​ ​ determine, based on the prediction value corresponding to the target pixel point of the first intermediate feature map and the first prediction value distribution rule corresponding to the first region, a prediction value corresponding to each pixel point in the first region; the first prediction value distribution rule indicates that the farther a pixel point is from the target pixel point of the first intermediate feature map in the first region, the smaller the prediction value corresponding to the pixel point is; determine, based on a second prediction value distribution rule corresponding to the second region, a prediction value corresponding to each pixel point in the second region; the second prediction value distribution rule indicates that the prediction value corresponding to each pixel point in the second region of the first intermediate feature map is smaller than the prediction value corresponding to each pixel point in the first region of the first intermediate feature map.

7. The method of claim 2, wherein, The method further comprises: determine a target pixel point of the first target feature map and a reference value corresponding to the target pixel point of the first target feature map from the L pixel points of the first target feature map, and determine a third region and a fourth region of the first target feature map based on the target pixel point of the first target feature map; the third region of the first target feature map includes the target pixel point of the first target feature map, and the fourth region of the first target feature map is other regions of the first target feature map except the third region; determine, based on the reference value corresponding to the target pixel point of the first target feature map and a first reference value distribution rule corresponding to the third region, a reference value corresponding to each pixel point in the third region; the first reference value distribution rule indicates that the farther a pixel point is from the target pixel point of the first target feature map in the third region, the smaller the reference value corresponding to the pixel point is; determine, based on a second reference value distribution rule corresponding to the fourth region, a reference value corresponding to each pixel point in the fourth region; the second reference value distribution rule indicates that the reference value corresponding to each pixel point in the fourth region of the first target feature map is smaller than the reference value corresponding to each pixel point in the third region of the first target feature map.

8. The method of claim 1, wherein, The determining a value of a contrast loss function based on a feature difference between the first target feature map and the second target feature map comprises: performing feature analysis processing on the first target feature map to obtain a first feature vector, and performing feature analysis processing on the second target feature map to obtain a second feature vector; performing operation on feature similarity between the first feature vector and the second feature vector by using a contrast loss function operation rule to obtain the value of the contrast loss function.

9. The method of claim 1, wherein, The respectively performing image classification processing on the first sample image and the second sample image based on the first target feature map and the second target feature map, and determining a value of a classification loss function according to a result of image classification processing and the first class label and the second class label comprises: performing feature analysis processing on the first target feature map to obtain a first feature vector, and performing feature analysis processing on the second target feature map to obtain a second feature vector; The first feature vector is classified and predicted to obtain a first category prediction result, and the second feature vector is classified and predicted to obtain a second category prediction result; The value of the classification loss function is determined based on the difference between the first category prediction result and the first category label and the difference between the second category prediction result and the second category label.

10. The method of claim 1, wherein, The image classification model comprises M convolution layers and N residual layers, and M is greater than or equal to 1; The image classification model is called to perform feature recognition processing on the first sample image to obtain a first feature map set, comprising: The M convolution layers in the image classification model are called to perform convolution processing on the first sample image to obtain a convolution feature map of the first sample image; The K residual layers in the image classification model are called to perform feature extraction processing on the convolution feature map of the first sample image to obtain the first intermediate feature map; The N-K residual layers in the image classification model are called to perform feature extraction processing on the first intermediate feature map to obtain the first target feature map.

11. The method of claim 1, wherein, After the image classification model is trained, the method further comprises: The M convolution layers in the trained image classification model are called to perform convolution processing on the to-be-processed image to obtain a convolution feature map of the to-be-processed image; The K residual layers in the trained image classification model are called to perform feature extraction processing on the convolution feature map of the to-be-processed image to obtain an intermediate feature map of the to-be-processed image; The N-K residual layers in the trained image classification model are called to perform feature extraction processing on the intermediate feature map of the to-be-processed image to obtain a target feature map of the to-be-processed image; The target feature map of the to-be-processed image is subjected to feature analysis processing to obtain a feature vector of the to-be-processed image; The feature vector of the to-be-processed image is subjected to classification and prediction processing to obtain a predicted category of the to-be-processed image.

12. An image processing apparatus characterized by comprising: Comprise: An acquisition unit is configured to acquire a training sample pair, wherein the training sample pair comprises a first sample image, a second sample image, a first category label corresponding to the first sample image, and a second category label corresponding to the second sample image; The processing unit is configured to invoke the image classification model to perform feature recognition processing on the first sample image to obtain a first feature map set, the first feature map set including a first intermediate feature map and a first target feature map, and to invoke the image classification model to perform feature recognition processing on the second sample image to obtain a second feature map set, the second feature map set including a second intermediate feature map and a second target feature map; the intermediate feature map is obtained from K residual layers, and the target feature map is obtained from N-K residual layers; pixel point information of the first feature map set includes predicted values corresponding to L pixel points in the first intermediate feature map and reference values corresponding to L pixel points in the first target feature map; pixel point information of the second feature map set includes predicted values corresponding to L pixel points in the second intermediate feature map and reference values corresponding to L pixel points in the second target feature map, L is an integer greater than or equal to 1, N is greater than or equal to 2, and K is greater than or equal to 1 and less than or equal to N; The processing unit is further configured to determine a first point loss value of a point loss function based on the predicted values corresponding to the L pixel points in the first intermediate feature map and the reference values corresponding to the L pixel points in the first target feature map, determine a second point loss value of the point loss function based on the predicted values corresponding to the L pixel points in the second intermediate feature map and the reference values corresponding to the L pixel points in the second target feature map, and perform summation operation on the first point loss value and the second point loss value to obtain a value of the point loss function. The processing unit is further configured to determine a value of a contrast loss function based on feature differences between the first target feature map and the second target feature map, perform image classification processing on the first sample image and the second sample image based on the first target feature map and the second target feature map respectively, and determine a value of a classification loss function based on results of the image classification processing and the first category label and the second category label. The processing unit is further configured to determine a value of a training loss function corresponding to the image classification model based on the value of the point loss function, the value of the contrast loss function, and the value of the classification loss function, and train the image classification model in a direction of reducing the value of the training loss function.

13. An image processing apparatus characterized by comprising: The image processing device includes an input interface and an output interface, and further includes: a processor adapted to implement one or more instructions; and a computer storage medium storing one or more instructions, the one or more instructions being adapted to be loaded and executed by the processor to implement the image processing method of any one of claims 1-11.

14. A computer storage medium, characterized in that, The computer storage medium stores computer program instructions, and the computer program instructions are executed by the processor to implement the image processing method of any one of claims 1-11.

15. A computer program product or computer program, characterised in that, The computer program product includes a computer program, and the computer program is executed by the processor to load and execute the image processing method of any one of claims 1-11.

Images