Image classification method, model training method, device, and storage medium

Abstract

Embodiments of the present application provide an image classification method, a model training method, a device and a storage medium. The image classification method comprises: obtaining a to-be-processed image and a category set used for analyzing and processing the to-be-processed image, the category set comprising a plurality of reference categories; determining an image feature of the to-be-processed image and a category set feature corresponding to the category set; determining a category context feature corresponding to the to-be-processed image based on the image feature and the category set feature, the category context feature comprising the image feature and at least part of the category set feature; and performing image processing based on the image feature and the category context feature to obtain a target category corresponding to the to-be-processed image. In the embodiments, the image processing based on the image feature and the category context feature can accurately perform image classification operation, especially for similar images, the target category of the image can also be accurately identified, and the practicability of the method is further improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image classification method, a model training method, an apparatus, and a storage medium. Background Technology

[0002] Currently, mainstream image classification networks are often trained based on the feedforward visual recognition paradigm, which involves first obtaining training images, then obtaining learning representations based on the training images, and finally training the visual classification network based on the learning representations to obtain the image classification network.

[0003] Although the image classification network obtained through the above method can accurately classify images of different categories, it cannot accurately classify similar images, thus failing to meet the requirements for more stringent image classification. Summary of the Invention

[0004] This application provides an image classification method, a model training method, an apparatus, and a storage medium that can accurately classify similar images and meet relatively strict image classification requirements.

[0005] In a first aspect, embodiments of this application provide an image classification method, including:

[0006] Obtain the image to be processed and a set of categories for analyzing and processing the image to be processed, the set of categories including multiple reference categories;

[0007] Determine the image features of the image to be processed and the category set features corresponding to the category set;

[0008] Based on the image features and the category set features, category context features corresponding to the image to be processed are determined, wherein the category context features include: the image features and at least a portion of the category set features;

[0009] Image processing is performed based on the image features and the category context features to obtain the target category corresponding to the image to be processed.

[0010] Secondly, embodiments of this application provide an image classification apparatus, including:

[0011] The first acquisition module is used to acquire an image to be processed and a category set for analyzing and processing the image to be processed, wherein the category set includes multiple reference categories;

[0012] The first determining module is used to determine the image features of the image to be processed and the category set features corresponding to the category set;

[0013] The first determining module is used to determine the category context features corresponding to the image to be processed based on the image features and the category set features, wherein the category context features include: the image features and at least a portion of the category set features;

[0014] The first processing module is used to perform image processing based on the image features and the category context features to obtain the target category corresponding to the image to be processed.

[0015] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor; wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions, when executed by the processor, implement the image classification method shown in the first aspect above.

[0016] Fourthly, embodiments of the present invention provide a computer storage medium for storing a computer program that, when executed by a computer, implements the image classification method described in the first aspect above.

[0017] Fifthly, embodiments of the present invention provide a computer program product, comprising: a computer program that, when executed by a processor of an electronic device, causes the processor to perform the steps in the image classification method described in the first aspect above.

[0018] Sixthly, embodiments of the present invention provide a method for training a network model, comprising:

[0019] Acquire training images, standard categories corresponding to the training images, and a category set for performing image classification operations, wherein the category set includes multiple reference categories;

[0020] Determine the image features of the training images and the category set features corresponding to the category set;

[0021] Based on the image features and the category set features, category context features corresponding to the training image are determined, wherein the category context features include: the image features and at least a portion of the category set features;

[0022] Based on the category context features, training images, and standard categories, a model training operation is performed to obtain a network model for implementing image processing operations.

[0023] In a seventh aspect, embodiments of the present invention provide a training apparatus for a network model, comprising:

[0024] The second acquisition module is used to acquire training images, standard categories corresponding to the training images, and a category set for implementing image classification operations, wherein the category set includes multiple reference categories;

[0025] The second determining module is used to determine the image features of the training image and the category set features corresponding to the category set;

[0026] The second determining module is used to determine the category context features corresponding to the training image based on the image features and the category set features, wherein the category context features include: the image features and at least a portion of the category set features;

[0027] The second processing module is used to perform model training operations based on the category context features, training images, and standard categories to obtain a network model for implementing image processing operations.

[0028] Eighthly, embodiments of this application provide an electronic device, including: a memory and a processor; wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions, when executed by the processor, implement the training method of the network model shown in the sixth aspect above.

[0029] Ninthly, embodiments of the present invention provide a computer storage medium for storing a computer program that, when executed by a computer, implements the training method for the network model described in the sixth aspect above.

[0030] In a tenth aspect, embodiments of the present invention provide a computer program product, comprising: a computer program that, when executed by a processor of an electronic device, causes the processor to perform the steps in the training method for the network model described in the sixth aspect above.

[0031] Eleventhly, embodiments of the present invention provide an image classification model, including:

[0032] The preprocessing layer is used to acquire the image to be processed and a set of categories for analyzing and processing the image to be processed, the set of categories including multiple reference categories; and to determine the image features of the image to be processed and the category set features corresponding to the set of categories.

[0033] The first-stage processing layer is used to downsample image features based on category set features to obtain downsampled features;

[0034] The second-stage processing layer, which may consist of one or more layers, is used to determine the category context features corresponding to the image to be processed based on the downsampled features and category set features. The category context features include the downsampled features and at least a portion of the category set features. The layer also generates a feature map corresponding to the image to be processed based on the downsampled features and category context features.

[0035] The classification layer is used to determine the target category corresponding to the image to be processed based on the feature map.

[0036] The image classification method, model training method, device, and storage medium provided in this application acquire the image to be processed and a set of categories for analyzing and processing the image to be processed. They determine the image features of the image to be processed and the category set features corresponding to the category set. Then, based on the image features and the category set features, they determine the category context features corresponding to the image to be processed. In this way, when performing image processing operations by combining image features and category context features, the target category corresponding to the image to be processed can be accurately obtained. Especially for similar images, the target category of the image can also be accurately determined, further improving the practicality of the method. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 A scene illustration of an image classification method provided in an embodiment of this application;

[0039] Figure 2 A flowchart illustrating an image classification method provided in an embodiment of this application;

[0040] Figure 3 This is a flowchart illustrating the process of determining the category context features corresponding to the image to be processed based on the image features and the category set features, as provided in an embodiment of this application.

[0041] Figure 4 A schematic diagram illustrating the principle of determining the category context features corresponding to the image to be processed, as provided in the embodiments of this application;

[0042] Figure 5 This is a flowchart illustrating the process of obtaining a target category corresponding to the image to be processed based on the image features and the category context features provided in this application embodiment.

[0043] Figure 6 A flowchart illustrating another image classification method provided in an embodiment of this application;

[0044] Figure 7 This is a flowchart illustrating the process of using the network model and the weight offset to process the image features and the category context features to obtain the target category of the image to be processed, as provided in this embodiment of the application.

[0045] Figure 8 A schematic diagram illustrating the principle of an image classification method provided in an application embodiment of this application;

[0046] Figure 9 A schematic diagram of a network model provided for an application embodiment of this application;

[0047] Figure 10 A schematic diagram illustrating the principle of obtaining weight offsets and sampling offsets corresponding to the network model, provided for application embodiments of this application;

[0048] Figure 11 A flowchart illustrating a network model training method provided in an embodiment of this application;

[0049] Figure 12 This is a schematic diagram of the structure of an image classification device provided in an embodiment of this application;

[0050] Figure 13 To and Figure 12 A schematic diagram of the electronic device corresponding to the image classification device provided in the illustrated embodiment;

[0051] Figure 14 A schematic diagram of the structure of a network model training device provided in an embodiment of this application;

[0052] Figure 15 To and Figure 14 The illustrated embodiment provides a schematic diagram of the electronic device corresponding to the network model training device. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0054] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the application. The singular forms "a," "the," and "the" used in the embodiments of this application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. "Multiple" generally includes at least two, but does not exclude the inclusion of at least one.

[0055] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0056] Depending on the context, the words “if” or “suppose” as used here can be interpreted as “when” or “in response to determination” or “in response to detection.” Similarly, depending on the context, the phrases “if determination” or “if detection (of the stated condition or event)” can be interpreted as “when determination” or “in response to determination” or “when detection (of the stated condition or event)” or “in response to detection (of the stated condition or event).”

[0057] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a product or system comprising a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a product or system. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the product or system that includes that element.

[0058] Furthermore, the timing of the steps in the following method embodiments is merely an example and not a strict limitation.

[0059] Terminology definition:

[0060] Deep learning refers to a collection of algorithms that use various machine learning algorithms on multi-layered neural networks to solve various problems such as images and text. While deep learning can be broadly categorized as a neural network, its implementation varies considerably. The core of deep learning is feature learning, which aims to acquire hierarchical feature information through layered networks, thereby solving the significant challenge of manually designing features in the past.

[0061] Artificial Neural Networks (ANNs) are a research hotspot in the field of artificial intelligence since the 1980s. They are computational models that abstract and build upon the neural networks of the human brain from an information processing perspective, forming different networks based on different connection methods.

[0062] Categorical context refers to the small number of candidate categories with high probability among all candidate categories for a specific input image / video. For example, if there are 400 candidate categories, the input will have a higher probability in 50 of them (obtained by a ranking method). These 50 categories serve as the categorical context information for that input.

[0063] To facilitate understanding of the technical solutions provided in the embodiments of this application by those skilled in the art, the relevant technologies are briefly described below: Currently, mainstream image classification networks are often trained based on the forward visual recognition paradigm, that is, training images are first obtained, learning representations are obtained based on the training images, and training operations of the visual classification network are performed based on the learning representations to obtain the image classification network.

[0064] However, because image classification networks tend to extract common features that are helpful for classifying most categories, and pay less attention to fine-grained features that are only useful for a few categories, it is difficult to identify fine-grained differences between images of similar categories. Thus, when using image classification networks for image classification operations, they can accurately classify images of different categories, but cannot accurately classify similar images, making it difficult to meet more stringent image classification requirements.

[0065] To address this problem, this embodiment proposes an image classification method, a model training method, an image classification model, and a device, wherein, with reference to the appendix... Figure 1As shown, the execution entity of the image classification method is an image classification device, which can be implemented as a client or a server. When the image classification device is implemented as a client, the client can communicate and connect with the server. Specifically, the client can be any programmable computing device with certain image classification capabilities. In specific implementations, the client can be a mobile phone, personal computer, tablet computer, smart wearable device, etc. Furthermore, the basic structure of the client can include at least one processor. The number of processors depends on the client's configuration and type. The client can also include memory, which can be volatile, such as RAM, or non-volatile, such as read-only memory (ROM), flash memory, etc., or both types can be included simultaneously. The memory typically stores the operating system (OS), one or more applications, and can also store program data. In addition to the processing unit and memory, the client also includes some basic configurations, such as a network card chip, I / O bus, display components, and some peripheral devices. Optionally, some peripheral devices can include, for example, a keyboard, mouse, input pen, printer, etc. Other peripheral devices are well known in the art and will not be described in detail here.

[0066] In other instances, the image classification device can be implemented as a server. A server is a device that can provide image classification services in a network virtual environment, typically referring to a device that utilizes a network for information planning and image classification operations. Physically, a server can be any device capable of providing computing services, responding to image classification requests, and displaying pages based on those requests. Examples include cluster servers, conventional servers, cloud servers, cloud hosts, and virtual data centers. The main components of a server include a processor, hard drive, memory, and system bus, similar to a general computer architecture.

[0067] In this case, the image classification method can be executed in the cloud, where multiple computing nodes (cloud servers) can be deployed. Each computing node has processing resources such as computing and storage. In the cloud, multiple computing nodes can be organized to provide a certain service; conversely, a single computing node can provide one or more services. The cloud can provide this service through an external service interface, which users can call to use the corresponding service. Service interfaces can take the form of Software Development Kits (SDKs) or Application Programming Interfaces (APIs).

[0068] According to the solution provided in this embodiment of the invention, the cloud can provide a service interface for image classification services. Users invoke this image classification service interface through an image classification device to trigger a request to the cloud to invoke the image classification service interface. The cloud determines the computing node for the image classification request and utilizes the processing resources in that computing node to perform the specific processing operations for image classification.

[0069] In the above embodiment, when the image classification device is implemented as a server, the server can connect to the client via a network, which can be a wireless or wired network connection. If the client and server are connected via a communication connection, the mobile network standard can be any one of 3G (WCDMA, TD-SCDMA, CDMA2000, UTMS), 4G (LTE), 4G+ (LTE+), WiMax, 5G, 6G, etc.

[0070] In this embodiment of the application, when the image classification device is implemented as a server, the server communicates with the client. The client is used to generate an image classification request, which may include an image to be processed. In order to realize the image classification operation, the client may actively or passively send the image to be processed to the image classification device, so that the image classification device can obtain the image to be processed.

[0071] An image classification device is used to acquire an image to be processed. To perform image classification, it acquires a set of categories for analyzing the image. This set of categories may include multiple reference categories, for example, 1000 reference categories. After acquiring the image to be processed and the set of categories, they can be processed separately to determine the image features of the image to be processed and the category set features corresponding to the category set. To accurately classify similar images, the image features and category set features can be analyzed to obtain category context features corresponding to the image to be processed. These category context features may include the image features and at least a portion of the category set features. After acquiring the image features and category context features, image processing operations can be performed based on these features to obtain the target category corresponding to the image to be processed.

[0072] The image classification method, model training method, image classification model, and device provided in this embodiment break through the limitations of the forward visual recognition paradigm. Specifically, by combining category context features and image features for image processing, fine-grained visual features corresponding to the image to be processed can be obtained. Then, the target category corresponding to the image to be processed can be obtained based on the fine-grained visual features, realizing image classification operation. Especially for similar images, it can also accurately classify similar images, further improving the accuracy and reliability of image classification operation and further improving the practicality of the method.

[0073] The following detailed description of some embodiments of the present invention is provided in conjunction with the accompanying drawings. Where there is no conflict between the embodiments, the following embodiments and features can be combined with each other. Furthermore, the timing of the steps in the following method embodiments is merely an example and not a strict limitation.

[0074] Figure 2 A flowchart illustrating an image classification method provided in this application embodiment; see attached document. Figure 2 As shown, this embodiment provides an image classification method. The execution subject of this method is an image classification device. It can be understood that the image classification device can be implemented as software, or a combination of software and hardware. Specifically, when the image classification device is implemented as hardware, it can be various electronic devices with image classification operations. When the image classification device is implemented as software, it can be installed in the electronic devices exemplified above. Specifically, the image classification method may include:

[0075] Step S201: Obtain the image to be processed and a set of categories for analyzing and processing the image to be processed. The set of categories includes multiple reference categories.

[0076] Step S202: Determine the image features of the image to be processed and the category set features corresponding to the category set.

[0077] Step S203: Based on image features and category set features, determine the category context features corresponding to the image to be processed. The category context features include: image features and at least a portion of the category set features.

[0078] Step S204: Perform image processing based on image features and category context features to obtain the target category corresponding to the image to be processed.

[0079] The specific implementation process and effects of each of the above steps are explained in detail below:

[0080] Step S201: Obtain the image to be processed and a set of categories for analyzing and processing the image to be processed. The set of categories includes multiple reference categories.

[0081] When a user has an image classification requirement, the image classification device can acquire the image to be processed. The number of images to be processed can be one or more. In some instances, the image to be processed can be stored in a third device. In this case, the image to be processed can be obtained through the third device (e.g., a client). Specifically, acquiring the image to be processed can include: identifying a third device that is communicatively connected to the image classification device, and actively or passively acquiring the image to be processed through the third device. The number of images to be processed can be one or more.

[0082] In other instances, the image to be processed can be obtained not only through a third device, but also through the video to be processed. In this case, obtaining the image to be processed may include: obtaining the video to be processed. Specifically, the video to be processed can be obtained from a preset storage area or a preset video shooting device (including: camera, camcorder, etc.), which can be pre-generated or generated in real time. Keyframes of the video to be processed are extracted to obtain the image to be processed corresponding to the video to be processed.

[0083] In the process of acquiring the image to be processed, or afterward, to accurately perform image classification, a category set for analyzing the image can be obtained. This category set may include multiple reference categories, such as "gray wolf," "husky," "Alaskan Malamute," "Siberian Husky," and "white wolf." In some instances, the category set can be obtained from a pre-configured database, or the reference categories in the category set can be obtained through human-computer interaction, or the category set can be obtained through analysis using a language knowledge base. In this case, obtaining the category set for analyzing the image to be processed may include: obtaining category text information to define the reference categories; and using a language knowledge base to analyze and recognize the category text information to obtain the reference categories included in the category set.

[0084] Step S202: Determine the image features of the image to be processed and the category set features corresponding to the category set.

[0085] After acquiring the image to be processed, it can be analyzed and processed to obtain the image features of the image to be processed. In some instances, determining the image features of the image to be processed may include: acquiring a machine learning model or image feature extraction algorithm for implementing the image feature extraction operation, and using the machine learning model or image feature extraction algorithm to process the image to be processed to obtain the image features of the image to be processed.

[0086] Similarly, after obtaining the category set, the category set can be analyzed and processed to obtain the category set features corresponding to the category set. In some instances, the category set features can be obtained through a pre-trained machine learning model or neural network model. In this case, determining the category set features corresponding to the category set can include: obtaining a pre-trained machine learning model or neural network model, inputting the category set into the machine learning model or neural network model, and obtaining the category set features output by the machine learning model or neural network model.

[0087] In other instances, category set features can be obtained not only through machine learning models or neural network models, but also by processing each reference category in the category set separately. In this case, determining the category set features corresponding to the category set may include: obtaining the reference category features of each reference category included in the category set. The specific method of obtaining the reference category features is similar to the specific method of obtaining the image features described above, and can be referred to the above description for details, which will not be repeated here.

[0088] After obtaining the reference category features for each reference category, they are integrated with the reference category features of all reference categories. In some instances, the integration operations such as concatenation and combination of all reference category features can be performed. For example, the reference category features include... By combining all the above reference category features, we can obtain and determine the category set features corresponding to the category set. The category set features can be:

[0089] Step S203: Based on image features and category set features, determine the category context features corresponding to the image to be processed. The category context features include: image features and at least a portion of the category set features.

[0090] In order to accurately classify similar images, after obtaining image features and category set features, the image features and category set features can be analyzed and processed to determine the category context features corresponding to the image to be processed. The category context features can include image features and at least part of the category set features. Since the category context features include both reference category features similar to the image to be processed and reference category features matching the image to be processed, the target category of the image to be processed can be accurately obtained when performing image classification based on the category context features.

[0091] Specifically, this embodiment does not limit the specific method for determining category context features. In some instances, category context features can be obtained through a pre-trained machine learning model or neural network model. In this case, determining the category context features corresponding to the image to be processed based on image features and category set features may include: obtaining a pre-trained machine learning model or neural network model for determining category context features, inputting image features and category set features into the machine learning model or neural network model, and obtaining the category context features output by the machine learning model or neural network model.

[0092] Step S204: Perform image processing based on image features and category context features to obtain the target category corresponding to the image to be processed.

[0093] After obtaining image features and category context features, image processing operations can be performed based on these features to obtain the target category corresponding to the image to be processed. In some instances, image processing operations can be performed using machine learning models or neural network models. In this case, obtaining the target category corresponding to the image to be processed based on image features and category context features can include: obtaining a pre-trained machine learning model or neural network model, inputting the image features and category context features into the machine learning model or neural network model, and obtaining the target category corresponding to the image to be processed output by the machine learning model or neural network model.

[0094] The image classification method provided in this embodiment obtains the image to be processed and a set of categories for analyzing and processing the image to be processed, determines the image features of the image to be processed and the category set features corresponding to the category set, and then determines the category context features corresponding to the image to be processed based on the image features and the category set features. In this way, when performing image processing operations by combining image features and category context features, the target category corresponding to the image to be processed can be accurately obtained. Especially for similar images, the target category of the image can also be accurately determined, which further improves the practicality of the method and is conducive to market promotion and application.

[0095] Figure 3 This application provides a flowchart illustrating the process of determining the category context features corresponding to the image to be processed based on image features and category set features, as shown in the attached diagram. Based on the above embodiments, refer to the appendix... Figure 3As shown, for category context features, not only can they be obtained by analyzing and processing image features and category set features using machine learning models or neural network models as described in the above embodiments, but they can also be determined by combining the various image processing stages used to analyze and process the image to be processed. In this case, determining the category context features corresponding to the image to be processed based on image features and category set features in this embodiment may include:

[0096] Step S301: Project the image features onto a preset embedding space to obtain the projected features.

[0097] To obtain the category context features corresponding to the image to be processed, after obtaining the image features, the image features can be projected onto a preset embedding space to obtain projected features. Specifically, projecting image features onto a preset embedding space to obtain projected features can include: performing global average pooling, non-linear processing, and fully connected processing operations on the image features to obtain projected features.

[0098] Step S302: Based on the category set features, determine the associated category features corresponding to the image to be processed in the non-first image processing stage.

[0099] Image classification often involves multiple processing stages, such as four, five, or six. It's important to note that the first processing stage (the initial stage) analyzes the entire set of category features and therefore lacks corresponding category context features. However, subsequent processing stages (the second, third, fourth, etc.) utilize parameters obtained after the first stage and thus possess corresponding category context features.

[0100] Since category context features are often related to the associated category features corresponding to non-first-time image processing stages, in order to accurately determine the category context features corresponding to non-first-time image processing stages, after obtaining the category set features, these features can be analyzed and processed to obtain and determine the associated category features corresponding to the image to be processed in the non-first-time image processing stages. In some instances, associated category features can be obtained through a pre-trained machine learning model. In this case, determining the associated category features corresponding to the image to be processed in the non-first-time image processing stages based on the category set features can include: obtaining a pre-trained machine learning model, inputting the category set features, the image to be processed, and the parameters of the image processing stage corresponding to the image to be processed into the machine learning model, and obtaining the associated category features corresponding to the image to be processed in the non-first-time image processing stages.

[0101] In other instances, the associated category features can be obtained through the associated category parameters corresponding to each image processing stage. In this case, determining the associated category features corresponding to the image to be processed in non-first-time image processing stages based on the category set features may include: determining the associated category parameters corresponding to each non-first-time image processing stage, wherein the associated category parameters corresponding to different image processing stages are the same or different; and determining the associated category features corresponding to the image to be processed in non-first-time image processing stages based on the associated category parameters from the category set features.

[0102] Specifically, the analysis and processing of the image to be processed may include multiple image processing stages. Different images to be processed may correspond to different numbers of image processing stages. In order to ensure the quality and effect of the image classification operation, the association category parameter corresponding to the non-first image processing stage can be determined. The association category parameter can be a parameter used to limit the number of association categories. For example, when the non-first image processing stage corresponding to the image to be processed includes a second image processing stage, a third image processing stage, and a fourth image processing stage, the association category parameter of the second image processing stage can be N2, the association category parameter of the third image processing stage can be N3, and the association category parameter of the fourth image processing stage can be N4. It should be noted that the association category parameters corresponding to different image processing stages may be the same or different, that is, N2 = N3 = N4, or N2 ≠ N3 ≠ N4.

[0103] In some instances, when the associated category parameters differ at different image processing stages, the associated category parameters can decrease as the image processing stage changes when analyzing the same image. Specifically, when N2≠N3≠N4, N2>N3>N4, meaning that the parameters decrease as the image processing stage progresses. This not only ensures the quality and efficiency of image analysis but also minimizes the computational resources required for image processing operations.

[0104] After obtaining the association category parameters corresponding to each non-first-time image processing stage, the association category features corresponding to the image to be processed in the non-first-time image processing stage can be determined based on the association category parameters in the category set features. In some instances, when the association category parameters corresponding to different image processing stages are different, determining the association category features corresponding to the image to be processed in the non-first-time image processing stage based on the association category parameters in the category set features may include: determining the similarity between each reference category in the category set corresponding to the previous image processing stage and the image to be processed; selecting the association category that satisfies the association category parameters based on the similarity in the category set corresponding to the previous image processing stage; and determining all category features corresponding to the association category in the category set features as the association category features corresponding to the image to be processed in the non-first-time image processing stage.

[0105] For example, N1 = 1000, N2 = 500, N3 = 200, N4 = 50. After obtaining the image to be processed and the category set, which includes 1000 reference categories, in the first image processing stage, the category set including the 1000 reference categories and the image to be processed can be analyzed to obtain the stage image features output in the first image processing stage. In order to accurately perform image processing operations in the second image processing stage, the similarity between the 1000 reference categories and the image to be processed or the stage image features can be obtained respectively, and the 1000 reference categories can be sorted based on the similarity. Then, N2 can be selected from the 1000 reference categories. (i.e., 500) associated categories. Then, all category features corresponding to the N2 associated categories can be determined from the category set features as the associated category features corresponding to the image to be processed in the non-first image processing stage. Referring to the image processing operations of the first and second image processing stages, the associated category features corresponding to the third image processing stage can be obtained sequentially based on the stage image features of the second image processing stage, and the associated category features corresponding to the fourth image processing stage can be obtained based on the stage image features corresponding to the third image processing stage, and so on. This effectively ensures the determination of the associated category features corresponding to the image to be processed in the non-first image processing stage.

[0106] It should be noted that the values ​​of N2, N3, and N4 are not limited to the values ​​listed above. Those skilled in the art can adjust the parameters arbitrarily according to specific application scenarios and requirements. In some examples, the value range of N2 can be (380, 600), the value range of N3 can be (150, 250), the value range of N4 can be (25, 75), and so on.

[0107] In some other instances, to avoid degrading image processing quality due to misselection of associated category features, when the associated category parameters are the same across different image processing stages, the method in this embodiment, based on the associated category parameters, to determine the associated category features corresponding to the image to be processed in the category set features in non-first-time image processing stages, may include: determining the similarity between each reference category in the category set and the image to be processed; reordering all reference categories in the category set based on the similarity to obtain a reordered category set; in the reordered category set, determining all reference categories ranked before a preset position as associated categories; and in the category set features, determining all category features corresponding to the associated categories as the associated category features corresponding to the image to be processed in non-first-time image processing stages.

[0108] Specifically, since image processing operations can include multiple processing stages, if the associated category features of the next processing stage are based on those of the previous stage, errors in the previous stage can easily propagate, thus reducing the quality and effectiveness of image processing. Therefore, to avoid propagating errors or amplifying their impact on image processing, the associated category parameters for different processing stages can be made identical. In this case, regardless of the number of processing stages used to analyze and process the image, the corresponding category sets will be the same for each stage.

[0109] For example, N1 = 1000, N2 = 1000, N3 = 1000, N4 = 1000. After obtaining the image to be processed and the category set, which includes 1000 reference categories, in the first image processing stage, the category set including the 1000 reference categories and the image to be processed can be analyzed and processed to obtain the stage image features output by the first image processing stage. In order to accurately obtain the image processing operation corresponding to the second image processing stage, the correlation between the 1000 reference categories and the image to be processed or the stage image features can be obtained. The similarity is calculated, and 1000 reference categories are reordered based on the similarity. Then, all reference categories that are ranked before a preset position (e.g., 500th, 600th, etc.) in the sorting are identified as associated categories. Then, all category features corresponding to the associated categories in the category set features are identified as associated category features corresponding to the image to be processed in the non-first image processing stage. In this way, in the second image processing stage, the associated category features and stage image features can be analyzed and processed, completing the second stage of image processing.

[0110] Similarly, before the third image processing stage, the obtained category set still includes 1000 reference categories. Then, the similarity between each of the 1000 reference categories and the features of the image to be processed or the stage image output from the second stage can be obtained. Based on the similarity, the 1000 reference categories are reordered. Then, all reference categories that are ranked before a preset position (e.g., 500th, 600th, 200th, 300th, etc.) can be identified as associated categories. Then, all category features corresponding to the associated categories in the category set features can be identified as associated category features corresponding to the image to be processed in the third image processing stage. In this way, in the third image processing stage, the associated category features and stage image features can be analyzed and processed, completing the third stage of image processing. By analogy, the image processing operations of each stage on the image to be processed can be completed.

[0111] Step S303: Based on the associated category features and the projected features, determine the category context features corresponding to the image to be processed.

[0112] After obtaining the associated category features and projected features, these features can be analyzed and processed to determine the category context features corresponding to the image to be processed. In some instances, the category context features can be obtained through a pre-trained machine learning model or neural network model. In this case, determining the category context features corresponding to the image to be processed based on the associated category features and projected features may include: obtaining a pre-trained machine learning model or neural network model, inputting the associated category features and projected features into the machine learning model or neural network model, and obtaining the category context features output by the machine learning model or neural network model.

[0113] In other instances, category context features can be obtained not only through pre-trained machine learning models or neural network models, but also directly by processing associated category features and projected features. In this case, determining the category context features corresponding to the image to be processed based on associated category features and projected features may include: integrating associated category features and projected features to obtain integrated features; and performing non-linear processing and pooling processing on the integrated features to obtain the category context features corresponding to the image to be processed.

[0114] For details, please refer to the appendix. Figure 4 As shown, after obtaining the associated category features and projected features, the associated category features and projected features can be integrated to obtain integrated features. After obtaining the integrated features, nonlinear processing, pooling processing, and other nonlinear processing operations can be performed on the integrated features. Among them, nonlinear processing operations can include fully connected processing (FC), layer normalization (LN), and rectified linear unit (ReLU). This can obtain the category context features corresponding to the image to be processed.

[0115] In this embodiment, by obtaining the projected features, the associated category features corresponding to the image to be processed in the non-first image processing stage are determined based on the category set features. Then, the category context features corresponding to the image to be processed are determined based on the associated category features and the projected features. This effectively ensures the accuracy and reliability of determining the category context features, and further improves the quality and effect of image processing operations based on category context features.

[0116] Figure 5 This application provides a flowchart illustrating image processing based on image features and category context features to obtain a target category corresponding to the image to be processed; based on the above embodiments, refer to the appendix... Figure 5 As shown, when performing image processing operations based on image features and category context, the image processing operations can be achieved not only through machine learning models or neural network models, but also by combining convolutional weights in the network model. In this case, obtaining the target category corresponding to the image to be processed based on image features and category context features can include:

[0117] Step S501: Based on image features and category context features, determine the weight offsets used to adjust the convolution weights in the network model, which is used to perform image classification operations.

[0118] When analyzing and processing images, to accurately classify different categories and similar images within the same category, it is necessary to detect fine-grained differences between them. This detection is often related to the convolutional weights in the network model. Therefore, to detect these fine-grained differences, after obtaining image features and category context features, these features can be analyzed to obtain weight shifts for adjusting the convolutional weights in the network model. The network model can be a pre-trained model for image classification, or it can be a model that requires training to perform image classification.

[0119] Specifically, this embodiment does not limit the specific method for determining the weight offset. In some instances, the weight offset can be obtained through a pre-trained machine learning model. In this case, determining the weight offset for adjusting the convolution weights in the network model based on image features and category context features may include: obtaining a pre-trained machine learning model, inputting image features and category context features into the machine learning model, and obtaining the weight offset output by the machine learning model.

[0120] In other instances, weight shifts can be obtained not only through a pre-trained machine learning model, but also through non-linear processing of image features and category context features. In this case, determining the weight shifts used to adjust the convolution weights in the network model based on image features and category context features may include: merging image features and category context features to obtain merged features; and performing non-linear processing on the merged features to obtain weight shifts.

[0121] Specifically, after obtaining image features and category context features, these features can be merged to obtain merged features. In some instances, image features and category context features can be directly combined to obtain merged features. In other instances, since image features are often three-dimensional while category context features are often one-dimensional, merging the image features and category context features to obtain merged features can be performed to ensure the quality and effectiveness of the merging process. This can include: pooling the image features to obtain pooled features with the same dimension as the category context features; and merging the pooled features and category context features to obtain merged features.

[0122] After obtaining the merged features, nonlinear processing can be applied to them to obtain weight shifts. In some instances, nonlinear processing algorithms can be directly used to process the merged features, and the result is determined as the weight shift. Alternatively, in other instances, nonlinear processing of the merged features to obtain weight shifts can include: performing fully connected processing on the merged features to obtain processed information; normalizing and linearly rectifying the processed information to obtain rectified information; and then performing fully connected processing on the rectified information to obtain the weight shift. This effectively ensures the accuracy and reliability of obtaining the weight shifts.

[0123] Step S502: Process the image features and category context features using the network model and weight shift to obtain the target category of the image to be processed.

[0124] In some instances, processing image features and category context features using a network model and weight shifts to obtain the target category of the image to be processed can include: adjusting the network model based on weight shifts to obtain an adjusted network model, and using the adjusted network model to process image features and category context features, thereby obtaining the target category of the image to be processed.

[0125] In other instances, processing image features and category context features using network models and weight shifts to obtain the target category of the image to be processed may include: obtaining the preset convolution weights corresponding to the network model; obtaining context weights based on weight shifts and preset convolution weights; and processing image features and category context features using the network model corresponding to the context weights to obtain the target category of the image to be processed.

[0126] Specifically, since weight offsets can adjust the convolutional weights in a network model, in order to obtain the adjusted network model, after obtaining the weight offsets, the corresponding preset convolutional weights of the network model can be obtained. Then, the weight offsets and preset convolutional weights can be analyzed and processed to obtain the context weights. In some instances, obtaining the context weights based on the weight offsets and preset convolutional weights can include: determining the context weights as the sum of the weight offsets and the preset convolutional weights; or, determining the context weights as the product of the weight offsets and the preset convolutional weights.

[0127] It is important to note that the range of weight offset values ​​can differ depending on the method used to obtain the context weights. For example, when context weights = weight offsets + preset convolution weights, the range of weight offset values ​​is the first range; when context weights = weight offsets * preset convolution weights, the range of weight offset values ​​is the second range. The second range is often smaller than the first range, which ensures the image processing quality and efficiency of the network model.

[0128] After obtaining the context weights, the network model corresponding to the context weights can be used to process the image features and category context features, thereby obtaining the target type of the image to be processed. For example, the image feature at a certain position p in the image to be processed is x, the preset convolution weight is w(k), the weight offset can be Δw(k), the preset sampling grid g(k) of the network model, and the preset sampling parameters p of the network model. Then, the network model can first process the image feature x to obtain the feature map y(p) corresponding to the image to be processed. Specifically, Then, the feature map y(p) can be classified based on the category context features to obtain the target category of the image to be processed.

[0129] In other instances, when performing analysis and processing operations on the image to be processed, image processing operations can be performed not only based on weight offsets, but also in conjunction with sampling offsets used to adjust the sampling grid in the network model. In this case, processing image features and category context features using the network model and weight offsets to obtain the target category of the image to be processed may include: determining the sampling offset used to adjust the sampling grid in the network model based on image features and category context features; and processing image features and category context features using the network model, weight offsets, and sampling offsets to obtain the target category of the image to be processed.

[0130] In order to further ensure the classification of different and similar images and to flexibly adjust the sampling grid corresponding to the network model when analyzing and processing the images, after obtaining the image features and category context features, the image features and category context features can be analyzed and processed to determine the sampling offset used to adjust the sampling grid in the network model.

[0131] Specifically, this embodiment does not limit the specific method for determining the sampling offset. In some instances, the sampling offset can be obtained through a pre-trained machine learning model. In this case, determining the sampling offset for adjusting the sampling grid in the network model based on image features and category context features may include: obtaining a pre-trained machine learning model, inputting image features and category context features into the machine learning model, and obtaining the sampling offset output by the machine learning model.

[0132] In other instances, sampling offsets can be obtained not only through a pre-trained machine learning model, but also through non-linear processing of image features and category context features. In this case, determining the sampling offset for adjusting the sampling grid in the network model based on image features and category context features may include: merging image features and category context features to obtain merged features; and performing non-linear processing on the merged features to obtain the sampling offset.

[0133] Specifically, after obtaining image features and category context features, the image features and category context features can be merged to obtain merged features. In some instances, the image features and category context features can be directly combined to obtain merged features. Alternatively, in other instances, since image features are often three-dimensional while category context features are often one-dimensional, in order to ensure the quality and effect of the merging process, the image features and category context features are merged to obtain merged features. This can include: expanding the category context features to obtain expanded features with the same dimension as the image features; and merging the expanded features and image features to obtain merged features.

[0134] After obtaining the merged features, nonlinear processing can be applied to them to obtain the sampling offset. In some instances, nonlinear processing algorithms can be used to process the merged features, and the result of the nonlinear processing can be determined as the sampling offset. Alternatively, nonlinear processing of the merged features to obtain the sampling offset can include: performing convolution processing on the merged features to obtain processed information; normalizing and linearly rectifying the processed information to obtain rectified information; and performing convolution processing on the rectified information to obtain the sampling offset, thereby effectively ensuring the accuracy and reliability of obtaining the sampling offset.

[0135] After obtaining the weight shift and sampling shift, the image features and category context features can be processed using the image model, weight shift, and sampling shift to obtain the target type of the image to be processed. In some instances, processing image features and category context features using the network model, weight shift, and sampling shift to obtain the target category of the image to be processed can include: adjusting the network model based on the weight shift and sampling shift to obtain the adjusted network model, and then using the adjusted network model to process the image features and category context features to obtain the target category of the image to be processed.

[0136] In other instances, processing image features and category context features using network models, weight shifts, and sampling shifts to obtain the target category of the image to be processed may include: obtaining the preset convolution weights, sampling grid, and preset sampling parameters corresponding to the sampling grid corresponding to the network model; obtaining context weights based on weight shifts and preset convolution weights; obtaining the shifted parameters based on sampling shifts and preset sampling parameters; determining the shifted sampling grid corresponding to the shifted parameters; and processing the image features and category context features using the network model corresponding to the context weights and shifted sampling grid to obtain the target category of the image to be processed.

[0137] Specifically, since weight offset can adjust the convolution weights in the network model, and sampling offset can adjust the sampling grid in the network model, after obtaining the weight offset and sampling grid, the preset convolution weights and preset sampling parameters corresponding to the network model can be obtained. Then, the weight offset and preset convolution weights can be analyzed and processed to obtain the context weights. Similarly, the sampling offset and preset sampling parameters can be analyzed and processed to obtain the offset parameters. In some instances, determining the offset sampling grid corresponding to the offset parameters can include: determining the sum of the offset parameters, the sampling grid, and the preset sampling parameters; determining the sampling grid corresponding to the sum as the offset sampling grid; or, obtaining the sum of the sampling grid and the preset sampling parameters, determining the product value between the sum and the offset parameters, and determining the sampling grid corresponding to the product value as the offset sampling grid. It should be noted that the offset sampling grid can be a regular grid or an irregular grid.

[0138] After obtaining the context weights and offset sampling grid, the network model corresponding to the context weights and offset sampling grid can be used to process the image features and category context features to obtain the target category of the image to be processed. For example, the image feature at a certain position p in the image to be processed is x, the preset convolution weight is w(k), the weight offset can be Δw(k), the preset sampling grid corresponding to the network model is g(k), the sampling offset is Δg(k), and the network model has preset sampling parameters p. Then, the network model can first process the image feature x to obtain the feature map y(p) corresponding to the image to be processed. Specifically, Then, the feature map y(p) can be classified based on the category context features to obtain the target category of the image to be processed.

[0139] In this embodiment, the weight offset used to adjust the convolution weights in the network model is determined based on image features and category context features. Then, the image features and category context features are processed using the network model and weight offset to obtain the target category of the image to be processed. This effectively ensures the accuracy and reliability of the target category determination and further improves the practicality of the method.

[0140] Figure 6 A flowchart illustrating another image classification method provided in this application embodiment; based on the above embodiments, refer to the appendix. Figure 6 As shown, after obtaining the target category of the image to be processed, this embodiment may include an operation of optimizing the network model based on a loss function. At this time, the method in this embodiment may further include:

[0141] Step S601: Obtain the stage loss function corresponding to each image processing stage in the network model.

[0142] When using a network model to process an image, a stage loss function corresponding to each image processing stage in the network model can be obtained. Different stage loss functions may correspond to different image processing stages of the image. In some examples, obtaining the stage loss function corresponding to each image processing stage in the network model may include: obtaining the learnable parameters corresponding to the network model, determining the standard category corresponding to the image to be processed, determining the first classification confidence score corresponding to each context category and the second classification confidence score corresponding to the standard category; and determining the stage loss function based on the first classification confidence score, the second classification confidence score, and the learnable parameters. Specifically, the stage loss function can be obtained using the following formula:

[0143]

[0144] Among them, L i Let represent the stage loss function corresponding to each image processing stage, П be used to identify the exponential function, and y be the standard category (true value) corresponding to the image to be processed. For the learnable parameters corresponding to the network model, s i (y) represents the second classification confidence level corresponding to the standard category. The first classification confidence score corresponding to each context category. The context category corresponding to each image processing stage.

[0145] It should be noted that the stage loss function corresponding to each image processing stage can not only be obtained through the above formula, but also by analyzing and processing the relevant operating parameters of each image processing stage through a pre-trained machine learning model or neural network model. As long as the accuracy and reliability of obtaining the stage loss function can be guaranteed, it will not be elaborated here.

[0146] Step S602: Based on the stage loss function corresponding to all image processing stages, determine the model loss function corresponding to the network model.

[0147] After obtaining the stage loss function corresponding to all image processing stages, we can analyze and process the stage loss function corresponding to all image processing stages to obtain the model loss function corresponding to the network model.

[0148] In some instances, the weight information corresponding to the loss function of each stage can be obtained. Based on the weight information, the loss functions of each stage are weighted and summed to obtain the model loss function corresponding to the network model. For example, when the image processing stage includes four stages, the stage loss function corresponding to the first image processing stage is L1, the stage loss function corresponding to the second image processing stage is L2, the stage loss function corresponding to the third image processing stage is L3, and the stage loss function corresponding to the fourth image processing stage is L4. The model loss function is L = α1L1 + α2L2 + α3L3 + α4L4.

[0149] Since the fourth image processing stage is used to perform image classification operations and determine the target category of the image to be processed, which is highly related to the image classification task, in order to ensure the quality and effect of the image classification operation, α4 ​​> (α3 + α2 + α1) is specified above. In some other instances, α4 can be 1, α3 = α2 = α1 = α, in which case L = α(L1 + L2 + L3) + L4, 1 > α, and the value range of α can be (0.1, 0.18), for example, α can be 0.15.

[0150] Step S603: Optimize the network model based on the model loss function to obtain the optimized model.

[0151] After obtaining the model loss function, the network model can be optimized based on the model loss function to obtain the optimized model. This can improve and ensure the image processing quality and effect of the network model, and further improve the practicality of the method.

[0152] Figure 7 This application provides a flowchart illustrating the process of using a network model and weight shifts to process image features and category context features to obtain the target category of the image to be processed, based on the above embodiments. (Refer to the appendix...) Figure 7 As shown, when there are multiple images to be processed, this embodiment can utilize a network model to perform parallel processing on multiple images. Specifically, the network model and weight shifts are used to process image features and category context features to obtain the target category of the image to be processed, which may include:

[0153] Step S701: Determine the number of images to be processed.

[0154] Since there can be multiple images to be processed, in order to ensure the quality and efficiency of processing multiple images, the number of images to be processed can be determined. In some instances, the number of images can be determined based on user interaction. In this case, the configuration page for the number of images can be displayed first, and the execution operation entered by the user in the configuration page can be obtained. The number of images to be processed can be determined based on the execution operation. Alternatively, in other instances, the number of images can be obtained by scanning and counting the images to be processed.

[0155] Step S702: Reshape the image features and weight offsets based on the number of images to obtain batch image features and batch weight offsets. The feature dimension of the batch image features is the same as the feature dimension of the image features, and the feature dimension of the batch weight offset is the same as the feature dimension of the weight offset. Both the batch image features and the batch weight offset include the number of images.

[0156] When there are multiple images to be processed, since the sampling regions and convolution weights of different images may be different when processing multiple images in parallel, after obtaining the number of images, the image features and weight offsets can be reshaped based on the number of images to obtain batch image features and batch weight offsets.

[0157] For example, if the number of images is B, the input channel size of the images is C, the output channel size of the images is C', h is the height of the image to be processed, w is the width of the image to be processed, and k is the kernel size, when analyzing and processing a single image to be processed, the corresponding image features can be represented as X(1, C, h, w), and the weight offset can be represented as W(C', C, k, k). When batch processing multiple images to be processed, the above image features can be reshaped into batch image features X'(1, B*C, h, w), and the weight offset can be reshaped into batch weight offset W'(B*C', C, k, k).

[0158] It should be noted that the specific methods for obtaining batch image features and batch weight offsets are not limited to the above implementation methods. Those skilled in the art can also use other methods to obtain batch image features and batch weight offsets, as long as the feature dimension of the obtained batch image features is the same as the feature dimension of the image features, the feature dimension of the batch weight offset is the same as the feature dimension of the weight offset, and both the batch image features and the batch weight offset include the number of images. This will not be elaborated further here.

[0159] Step S703: Perform parallel processing on batch image features based on the network model and batch weight offset to obtain the target category corresponding to each image to be processed.

[0160] After obtaining the batch weight offsets and batch image features, the batch image features can be processed in parallel based on the network model and the batch weight offsets, thereby obtaining the target category corresponding to each image to be processed. It should be noted that when performing convolution processing on the batch image features based on the network model, the batch weight offsets W`(B*C`, C, k, k) can be divided into B groups first, and then the convolution processing operation can be performed in parallel based on the batch weight offsets of the B groups. This effectively improves the quality and efficiency of image processing.

[0161] In this embodiment, by determining the number of images to be processed, the image features and weight offsets are reshaped based on the number of images to obtain batch image features and batch weight offsets. Then, the batch image features are processed in parallel based on the network model and batch weight offsets to obtain the target category corresponding to each image to be processed. This effectively realizes the parallel processing operation of batch images and further improves the quality and efficiency of image processing.

[0162] In practical applications, to address the problem of "difficulty in capturing fine-grained differences between visual categories" in existing technologies, please refer to the appendix. Figure 8 As shown, this application embodiment provides an image classification method based on a novel computer vision foundational network. This method can utilize category context (top-k categories) to guide the extraction of visual features, and then perform multiple visual tasks such as image classification, instance segmentation, and video classification based on the extracted visual features. It achieves significant accuracy improvements in these tasks with relatively small additional computational costs. Specifically, taking a computer vision foundational network (hereinafter referred to as the network model) comprising four image processing stages as an example, the method can include the following steps:

[0163] Step 1: Obtain the image to be processed and the set of categories used to perform image classification operations. The set of categories includes K reference categories.

[0164] Step 2: Determine the input image features of the image to be processed and the category set features corresponding to the category set.

[0165] Among them, reference appendix Figure 9 As shown, the network model can include a preprocessing layer, which can be consistent with the design of a standard convolutional neural network. The stemming network in the network can use a 4×4 stride convolution, which can perform 4 downsampling operations on the input image, and the output has C=96 channel features. That is, after the preprocessing layer obtains the H×W×3 input RGB image, it can process the RGB image to obtain the corresponding input image features. In order to balance the computational cost of image processing and improve the efficiency of image processing, the input image features can be... in, H is the image resolution, W is the image height, and C is the image width. Similarly, the preprocessing layer can analyze and process the category set to obtain the category set features corresponding to the category set. The category set features can be represented as... The above R d For the real number field, or the aforementioned category set features can also be generated by an N1×d embedding layer. It should be noted that the features output by the preprocessing layer can be not only 96 channels, but also 128 channels, 256 channels, etc.

[0166] Step 3: Perform nonlinear processing on the input image features and category set features to obtain the weight offset and sampling offset corresponding to the network model. The weight offset is added to the static kernel weights of the network model, and the sampling offset is added to the regular grid sampling positions of the network model to improve the processing quality and effect of the network model.

[0167] After obtaining the input image features and category set features, the category context features corresponding to the image to be processed can be determined based on these features. Then, the weight offsets and sampling offsets corresponding to the network model can be determined based on the category context features and the input image features. For details, please refer to the appendix. Figure 10 As shown, since the feature dimension of the category context feature is d*1, and the feature dimension of the image feature can be c*h*w, pooling is performed on the image feature x to obtain the pooled feature, which has the same dimension as the category context feature. Then, the pooled feature and the category context feature are bridged to obtain the processed feature. The processed feature is then subjected to fully connected, normalized, and linearly rectified operations to obtain the rectified feature. Finally, a fully connected operation is performed on the rectified feature to obtain the weight offset Δw, where the size of Δw is configured as 1×c×7×7. The size of this weight offset is the same as the size of the preset convolution weight w in the network model, so that the weight offset Δw can adaptively adjust the convolution weights in the network model based on the context prior knowledge.

[0168] Similarly, please refer to the appendix. Figure 10As shown, after obtaining the category context features, the category context can be expanded to obtain expanded features. The dimensions of the expanded features are d×h×w. Then, the expanded features and the input image features x are bridged. Specifically, the expanded features and the input image features can be connected along the channel dimension to obtain the processed features (c+d)×h×w. The obtained processed features are then sent to a non-linear convolutional block for processing. This block consists of a 1×1 convolution (followed by LN and ReLU) and a 7×7 depthwise convolution. This allows for convolution, normalization, and linear rectification of the processed features to obtain rectified features. After obtaining the rectified features, convolution can be performed on them to obtain the sampling offset Δp. The output sampling offset has the same resolution as the input features, and this sampling offset Δp can achieve free-form deformation of the sampling grid corresponding to the network model.

[0169] Step 4: Based on image features and category set features, determine the category context features corresponding to the image to be processed. The category context features include: image features and at least a portion of the category set features.

[0170] The acquisition of category context features can include a classification process and a merging process. The classification process is used to reduce the number of categories considered from N to N in each image processing stage that analyzes and processes the image to be processed. i Reduce to N i+1 Where i is the index of the corresponding image processing stage, the merging process can merge N i+1 The embeddings of the considered classes are merged into a context prior for use in subsequent contextual convolutional blocks.

[0171] For details, please refer to the appendix. Figure 4 As shown, to compare the embedding vectors corresponding to the features of the category set, visual features x from each image processing stage are projected into the embedding space. This projection involves a global pooling layer, a non-linear processing layer, and two fully connected (FC) layers. The non-linear processing layer may include a global pooling layer, layer normalization, and a linear rectified function layer. The output of the projection is a projected feature (or visual feature vector x') of dimension d. Then, the projected features and N... i Calculate the cosine similarity between the features of the class categories, and then generate the similarity vector s. i As N i The loss is calculated based on the classification scores of the classes, with the highest score s being the highest. i The first N i+1 Category features are collected and propagated to the next stage of image processing.

[0172] Then, given N in the classification step i+1Class, N i+1 The category set features corresponding to each class are merged with the projected features to obtain the integrated embedding vector. The integrated embedding vector can then be transformed to obtain the category context features C (also known as context priors), C∈R. d Since the category context features summarize the similarities and differences between the considered classes, using them as task information in the following contextual convolutional blocks can extract more discriminatory features or fine-grained features.

[0173] Step 5: Obtain the static weights corresponding to the network model (corresponding to the preset weight information in the above embodiments), and determine the context weights based on the static weights and weight offsets.

[0174] Step 6: Obtain the preset sampling parameters corresponding to the network model, and determine the offset sampling grid based on the sampling offset and the preset sampling parameters.

[0175] Step 7: Use the network model corresponding to the context weights and offset sampling grids to process the input image features and category context features to obtain the output feature map corresponding to the image to be processed.

[0176] Among them, reference appendix Figure 9 As shown, the network model may also include four image processing layers that communicate with the preprocessing layer and a classification layer that communicates with the image processing layers. The four image processing layers may be the image processing layers corresponding to stage1, stage2, stage3 and stage4, respectively. The image processing layer corresponding to stage1 is different from the image processing layers corresponding to stage2, stage3 and stage4, while the image processing layers of stage2, stage3 and stage4 mentioned above are the same.

[0177] Specifically, the image processing layer corresponding to stage1 may include a standard convolutional module, which is used to process the input image features output by the preprocessing layer to obtain the stage output features. After obtaining the stage output features, the stage output features can be input into the image processing layer corresponding to stage 2. The image processing layer corresponding to stage 2 can include a contextual convolution module and a contextualization module. The contextualization module is used to obtain the input image features output by stage 1, and select the associated contextual features corresponding to the image processing layer of stage 2 based on the input image features in the image set corresponding to stage 1. In some instances, a pre-configured associated category parameter corresponding to the image processing layer of stage 2 can be obtained first. The associated category parameter can be N2. The category parameter corresponding to stage 1 can be the number of all reference categories corresponding to the category set, which can be N1. N2 < N1. For example, N1 is the number of all possible categories of the image. When N1 = 1000, N2 can be 500. At this time, the contextualization module can select N2 categories from N1 categories as the associated categories of stage 2 based on the stage output features, and determine the features corresponding to the associated categories as the associated contextual features.

[0178] After obtaining the associated context features, the associated context features and the stage output features can be input into the context convolution module for processing, thereby obtaining the stage output features corresponding to stage 2. These stage output features can be features that have undergone downsampling processing. Specifically, the stage output features can be... The output features of each stage can then be input into the image processing layer corresponding to stage 3 and stage 4. The image processing operations of stage 3 and stage 4 are similar to those of stage 2. For details, please refer to the above description and will not be repeated here.

[0179] It's important to note that the image processing operation in stage 3 corresponds to an associated category parameter of N3, and the image processing operation in stage 4 corresponds to an associated category parameter of N4. In this case, N4 < N3 < N2. In practice, N3 can be 200, N4 can be 50, etc. Furthermore, the output feature of stage 3 is the feature obtained by downsampling the output feature of stage 2. This output feature can be... Similarly, the stage 4 output feature is a feature obtained by downsampling the stage 3 output feature. This stage output feature can be... Specifically, the feature map resolution of "Phase 3" is Maintaining 4C channels and reducing the number of classes considered to N3, the feature map resolution for the "fourth stage" is... Maintaining 8C channels, the number of classes considered is reduced to N4. As the number of classes considered gradually decreases, the associated contextual features convey more fine-grained classification information, which correspondingly modulates higher-level contextual convolutions to extract more discriminative features.

[0180] Through the above process, it is possible to effectively extract the associated context features from the most likely image class. For example, for stage2, when performing image processing operations, the number of classes considered for implementing the image processing operations can be reduced from N1 to N2, and the embeddings of the N2 classes can be merged as associated context features. This can effectively improve the quality and efficiency of image processing.

[0181] After obtaining the associated contextual features corresponding to each image processing stage, the network model can perform image processing operations based on these features. For example, consider an image feature x at a certain location p in the image to be processed, with preset convolutional weights w(k) and weight offsets Δw(k). The network model has a preset sampling grid g(k) and a sampling offset Δg(k), and preset sampling parameters p. The network model can then process the image feature x to obtain the feature map y(p) corresponding to the image to be processed. Specifically...

[0182] Step 8: Perform image classification based on the output feature map to obtain the target category corresponding to the image to be processed.

[0183] After obtaining the output feature map, image classification can be performed on the output feature map based on the classification module. Specifically, the number of image categories can be reduced from N4 to 1, thereby obtaining the target category corresponding to the image to be processed, and completing the image classification operation.

[0184] For example, when processing an animal image of a "wolf," because "husky" and "wolf" have similar appearances, existing network models would likely identify the animal as a husky. However, the method in this application embodiment can predict the wolf's identity by using its long, upright ears (determined by the offset sampling area defined by weight offset and sampling offset). This allows for accurate identification of the animal in the image as a "wolf." Similarly, the method in this application embodiment can also accurately distinguish between a "space heater" and an "electric fan." Specifically, it can perform image recognition by identifying the differences between the base and the central hub of the object, thereby improving the quality and effectiveness of image classification.

[0185] It is important to note that the module used for feature extraction in this application embodiment is not limited to a specific form. It is suitable for any corresponding structure with the same function, such as two-dimensional convolutional layers, three-dimensional convolutional layers, and Transformer layers. The Transformer layer is a classic NLP (Natural Language Processing) model that encodes input and computes output entirely based on attention, without relying on sequence-aligned recurrent neural networks or convolutional neural networks. Furthermore, the category representation in this embodiment is not limited to a specific form and is suitable for any corresponding structure with the same function, such as k-hot encoded vectors, randomly initialized embedding vectors, and embedding vectors from a language model.

[0186] In addition to performing the image processing operations described above, the method in this embodiment can also perform optimization operations on the network model. Specifically, the method in this embodiment may include:

[0187] Step 11: Obtain the standard category corresponding to the image to be processed and the learnable parameters corresponding to the network model;

[0188] Step 12: Based on the standard categories, learnable parameters, and the stage loss function corresponding to each image processing stage;

[0189] The stage loss function can be obtained using the following formula:

[0190]

[0191] Among them, L i Let represent the stage loss function corresponding to each image processing stage, Π be used to identify the exponential function, and y be the standard category (true value) corresponding to the image to be processed. For the learnable parameters corresponding to the network model, s i(y) represents the second classification confidence level corresponding to the standard category. The first classification confidence corresponding to each context category can be obtained by calculating the cross-entropy loss function. This effectively allows the top N categories with relatively high confidence to be collected and propagated to the next image processing stage. The context category corresponding to each image processing stage.

[0192] Step 13: Based on all the stage loss functions, determine the model loss function corresponding to the network model.

[0193] The model loss function can be obtained by the following formula: L=α(L1+L2+L3)+L4, where α can be the weight information corresponding to the first-stage loss function, the second-stage loss function, and the third-stage loss function. For the weight information α, it can be 0.15, etc.

[0194] Step 14: Optimize the network model based on the model loss function to obtain the optimized model.

[0195] The technical solution provided in this application embodiment offers a contextual convolutional network that utilizes some of the most likely classes as contextual information for representation learning, thereby surpassing the performance of corresponding models on various visual tasks (including image classification, video classification, and object segmentation). This reveals the potential of contextual representation learning for computer vision. Specifically, this contextual convolutional network can provide image / video recognition capabilities, such as scene recognition in detection and control scenarios, and fight recognition; it can also provide high-quality pre-trained weights for downstream tasks (object detection, etc.).

[0196] Specifically, this technical solution predicts the top-k associated categories of the input based on the contextual category features obtained in the previous image processing stage. Then, it uses these top-k associated categories to adjust the sampling method and weights of the feature extraction layer to obtain new features. Specifically, the model's category context, i.e., the top-k categories, can be determined based on the output features of the previous layer, with k initially representing the number of all possible categories. During the image processing described above, the model's extracted feature semantics (i.e., the weights of the feature extraction layer) and region of interest (i.e., the sampling method of the feature extraction layer) can be adjusted based on the model's top-k associated categories. This achieves adaptive focus on fine-grained features helpful for top-k classification. By continuously repeating these steps, finer-grained features can be extracted and the number of top-k categories reduced. The process stops when k = 1, effectively achieving image classification.

[0197] Qualitative analysis shows that the method in this embodiment not only ensures the accuracy of image processing but also maximizes throughput, demonstrating the potential of context labeling. Furthermore, this method can be extended to general visual recognition tasks such as video classification and is also applicable to downstream intensive visual tasks (e.g., video segmentation, video detection, etc.), thereby effectively improving the flexibility, reliability, and applicability of the method and facilitating its market promotion and application.

[0198] Figure 11 A flowchart illustrating a network model training method provided in this application embodiment; see attached document. Figure 11 As shown, this embodiment provides a method for training a network model. The execution entity of this method is a network model training device. It can be understood that the network model training device can be implemented as software, or a combination of software and hardware. Specifically, when the network model training device is implemented as hardware, it can be various electronic devices with network model training operations. When the network model training device is implemented as software, it can be installed in the electronic devices exemplified above. Specifically, the network model training method may include:

[0199] Step S1101: Obtain the training image, the standard category corresponding to the training image, and the category set used to perform the image classification operation. The category set includes multiple reference categories.

[0200] When a user has a need for model training, they can obtain training images and a set of categories for image classification operations. Specifically, the method of obtaining training images is similar to the method of obtaining images to be processed in the above embodiments. The only difference is that the obtained training images correspond to standard categories. The category set may include multiple reference categories, and the multiple reference categories may include standard categories, or the multiple reference categories may not include standard categories. For details, please refer to the above statements, which will not be repeated here.

[0201] Step S1102: Determine the image features of the training images and the category set features corresponding to the category set.

[0202] Step S1103: Based on image features and category set features, determine the category context features corresponding to the training image. The category context features include: image features and at least a portion of the category set features.

[0203] The specific implementation methods and effects of steps S1102-S1103 in this embodiment are similar to those of steps S202-S203 in the above embodiment. For details, please refer to the above description, which will not be repeated here.

[0204] Step S1104: Perform model training based on category context features, training images, and standard categories to obtain a network model for image processing operations.

[0205] After obtaining the category context features, training images, and standard categories, the model can be trained based on these features to obtain a network model for image processing operations, which can perform image classification operations.

[0206] The network model training method provided in this embodiment obtains training images, standard categories corresponding to the training images, and a set of categories used to perform image classification operations. It then determines the image features of the training images and the category set features corresponding to the category set. Based on the image features and category set features, it determines the category context features corresponding to the training images. Finally, it performs model training based on the category context features, the training images, and the standard categories to obtain a network model for image processing operations. Since the network model is trained based on category context features, image classification operations can be performed in conjunction with the category context when performing image processing operations based on the network model. This improves the accuracy of image category recognition operations, further enhancing the practicality of the method and facilitating its market promotion and application.

[0207] Furthermore, this embodiment provides an image classification model, which may include:

[0208] The preprocessing layer is used to acquire the image to be processed and a set of categories for analyzing and processing the image to be processed, the set of categories including multiple reference categories; and to determine the image features of the image to be processed and the category set features corresponding to the set of categories.

[0209] The first-stage processing layer is used to downsample image features based on category set features to obtain downsampled features;

[0210] The second-stage processing layer, which may consist of one or more layers, is used to determine the category context features corresponding to the image to be processed based on the downsampled features and category set features. The category context features include the downsampled features and at least a portion of the category set features. The layer also generates a feature map corresponding to the image to be processed based on the downsampled features and category context features.

[0211] The classification layer is used to determine the target category corresponding to the image to be processed based on the feature map.

[0212] It should be noted that the specific implementation method and effect of the image classification model in this embodiment are the same as those described above. Figures 2-10The specific implementation and effect of the method shown are similar. For parts not described in detail in this embodiment, please refer to the [example / description]. Figures 2-10 The relevant descriptions of the embodiments shown will not be repeated here.

[0213] Figure 12 This is a schematic diagram of the structure of an image classification device provided in an embodiment of this application; see attached drawing. Figure 12 As shown, this embodiment provides an image classification device that can perform the above-described... Figure 2 The image classification method shown, specifically, the image classification device may include:

[0214] The first acquisition module 11 is used to acquire the image to be processed and a category set for analyzing and processing the image to be processed, the category set including multiple reference categories;

[0215] The first determining module 12 is used to determine the image features of the image to be processed and the category set features corresponding to the category set;

[0216] The first determining module 12 is used to determine the category context features corresponding to the image to be processed based on image features and category set features. The category context features include: image features and at least a portion of the category set features.

[0217] The first processing module 13 is used to perform image processing based on image features and category context features to obtain the target category corresponding to the image to be processed.

[0218] In some instances, when the first determining module 12 determines the category context features corresponding to the image to be processed based on image features and category set features, the first determining module 12 is used to perform: projecting image features onto a preset embedding space to obtain projected features; determining the associated category features corresponding to the image to be processed in non-first image processing stages based on category set features; and determining the category context features corresponding to the image to be processed based on associated category features and projected features.

[0219] In some instances, when the first determining module 12 determines the associated category features corresponding to the image to be processed in the non-first image processing stage based on the category set features, the first determining module 12 is used to perform: determining the associated category parameters corresponding to each non-first image processing stage, wherein the associated category parameters corresponding to different image processing stages are the same or different; and determining the associated category features corresponding to the image to be processed in the non-first image processing stage based on the associated category parameters.

[0220] In some instances, when the associated category parameters differ at different image processing stages, the associated category parameters decrease as the image processing stage changes.

[0221] In some instances, when the first determining module 12 determines the associated category features corresponding to the image to be processed in the category set features in the non-first image processing stage based on the associated category parameters, the first determining module 12 is used to perform: determining the similarity between each reference category in the category set corresponding to the previous image processing stage and the image to be processed; selecting the associated category that satisfies the associated category parameters based on the similarity in the category set corresponding to the previous image processing stage; and determining all category features corresponding to the associated category in the category set features as the associated category features corresponding to the image to be processed in the non-first image processing stage.

[0222] In some instances, when the associated category parameters are the same across different image processing stages, when the first determining module 12 determines the associated category features corresponding to the image to be processed in the non-first image processing stage based on the associated category parameters, the first determining module 12 performs the following: determining the similarity between each reference category in the category set and the image to be processed; reordering all reference categories in the category set based on the similarity to obtain a reordered category set; determining all reference categories ranked before a preset position in the reordered category set as associated categories; and determining all category features corresponding to the associated categories in the category set as associated category features corresponding to the image to be processed in the non-first image processing stage.

[0223] In some instances, when the first determining module 12 determines the category context features corresponding to the image to be processed based on the associated category features and the projected features, the first determining module 12 is used to perform: integrating the associated category features and the projected features to obtain integrated features; performing nonlinear processing and pooling processing on the integrated features to obtain the category context features corresponding to the image to be processed.

[0224] In some instances, when the first processing module 13 performs image processing based on image features and category context features to obtain the target category corresponding to the image to be processed, the first processing module 13 is used to perform: determining a weight offset for adjusting the convolution weights in the network model based on image features and category context features, the network model being used to implement image classification operations; and processing the image features and category context features using the network model and weight offset to obtain the target category of the image to be processed.

[0225] In some instances, when the first processing module 13 determines the weight offset for adjusting the convolution weights in the network model based on image features and category context features, the first processing module 13 performs the following: merging the image features and category context features to obtain merged features; and performing nonlinear processing on the merged features to obtain the weight offset.

[0226] In some instances, when the first processing module 13 performs a merging process on the image features and the category context features to obtain the merged features, the first processing module 13 is used to perform: pooling processing on the image features to obtain pooled features with the same dimension as the category context features; and merging the pooled features and the category context features to obtain the merged features.

[0227] In some instances, when the first processing module 13 performs nonlinear processing on the merged features to obtain weight offsets, the first processing module 13 is used to perform: fully connected processing on the merged features to obtain processed information; normalizing and linearly rectifying the processed information to obtain rectified information; and fully connected processing on the rectified information to obtain weight offsets.

[0228] In some instances, when the first processing module 13 processes image features and category context features using a network model and weight offsets to obtain the target category of the image to be processed, the first processing module 13 is used to perform: obtaining the preset convolution weights corresponding to the network model; obtaining the context weights based on the weight offsets and preset convolution weights; and processing the image features and category context features using the network model corresponding to the context weights to obtain the target category of the image to be processed.

[0229] In some instances, when the first processing module 13 obtains the context weights based on the weight offset and the preset convolution weights, the first processing module 13 is used to perform the following: determining the sum of the weight offset and the preset convolution weights as the context weights; or determining the product of the weight offset and the preset convolution weights as the context weights.

[0230] In some instances, when the first processing module 13 processes image features and category context features using a network model and weight offsets to obtain the target category of the image to be processed, the first processing module 13 is used to perform: determining a sampling offset for adjusting the sampling grid in the network model based on image features and category context features; and processing image features and category context features using the network model, weight offsets, and sampling offsets to obtain the target category of the image to be processed.

[0231] In some instances, when the first processing module 13 determines the sampling offset for adjusting the sampling grid in the network model based on image features and category context features, the first processing module 13 performs the following: merging the image features and category context features to obtain merged features; and performing nonlinear processing on the merged features to obtain the sampling offset.

[0232] In some instances, when the first processing module 13 merges the image features and the category context features to obtain the merged features, the first processing module 13 is used to perform: expanding the category context features to obtain expanded features with the same dimension as the image features; merging the expanded features and the image features to obtain the merged features.

[0233] In some instances, when the first processing module 13 performs nonlinear processing on the merged features to obtain a sampling offset, the first processing module 13 is used to perform: convolution processing on the merged features to obtain processed information; normalization and linear rectification processing on the processed information to obtain rectified information; and convolution processing on the rectified information to obtain a sampling offset.

[0234] In some instances, when the first processing module 13 processes image features and category context features using a network model, weight offset, and sampling offset to obtain the target category of the image to be processed, the first processing module 13 performs the following: obtaining the preset convolution weights, sampling grid, and preset sampling parameters corresponding to the sampling grid corresponding to the network model; obtaining the context weights based on the weight offset and preset convolution weights; obtaining the offset parameters based on the sampling offset and preset sampling parameters; determining the offset sampling grid corresponding to the offset parameters; and processing the image features and category context features using the network model corresponding to the context weights and offset sampling grid to obtain the target category of the image to be processed.

[0235] In some instances, when the first processing module 13 determines the offset sampling grid corresponding to the offset parameters, the first processing module 13 is used to perform: determining the sum of the offset parameters, the sampling grid, and the preset sampling parameters; determining the sampling grid corresponding to the sum as the offset sampling grid; or, obtaining the sum of the sampling grid and the preset sampling parameters, determining the product value between the sum value and the offset parameters, and determining the sampling grid corresponding to the product value as the offset sampling grid.

[0236] In some instances, after obtaining the target category of the image to be processed, the first acquisition module 11, the first determination module 12, and the first processing module 13 in this embodiment are used to perform the following steps:

[0237] The first acquisition module 11 is used to acquire the stage loss function corresponding to each image processing stage in the network model;

[0238] The first determining module 12 is used to determine the model loss function corresponding to the network model based on the stage loss function corresponding to all image processing stages;

[0239] The first processing module 13 is used to optimize the network model based on the model loss function to obtain the optimized model.

[0240] In some instances, when there are multiple images to be processed, when the first processing module 13 processes image features and category context features using a network model and weight shifts to obtain the target category of the image to be processed, the first processing module 13 performs the following: determining the number of images to be processed; reshaping image features and weight shifts based on the number of images to obtain batch image features and batch weight shifts, wherein the feature dimension of the batch image features is the same as the feature dimension of the image features, the feature dimension of the batch weight shifts is the same as the feature dimension of the weight shifts, and both the batch image features and batch weight shifts include the number of images; and performing parallel processing of the batch image features based on the network model and batch weight shifts to obtain the target category corresponding to each image to be processed.

[0241] Figure 12 The image classification device shown can perform Figures 1-9 For the methods shown in the embodiments, the parts not described in detail in this embodiment can be referred to the following: Figures 1-9 The relevant descriptions of the illustrated embodiments are provided below. For the execution process and technical effects of this technical solution, please refer to [link / reference]. Figures 1-9 The descriptions in the illustrated embodiments will not be repeated here.

[0242] In one possible design, Figure 12 The structure of the image classification device shown can be implemented as an electronic device. (See attached document.) Figure 13 As shown, the image classification device in this embodiment can be implemented as an electronic device, such as a mobile phone, tablet computer, personal computer, smart wearable device, server, or other device capable of image processing operations. Specifically, the electronic device may include a first processor 21 and a first memory 22. The first memory 22 is used to store the data executed by the corresponding electronic device. Figure 2 In the illustrated embodiment, the program for the image classification method is configured such that the first processor 21 is configured to execute the program stored in the first memory 22.

[0243] The program includes one or more computer instructions, wherein when executed by the first processor 21, the one or more computer instructions can perform the following steps: acquiring an image to be processed and a set of categories for analyzing and processing the image to be processed, the set of categories including multiple reference categories; determining image features of the image to be processed and category set features corresponding to the set of categories; determining category context features corresponding to the image to be processed based on the image features and the category set features, the category context features including: image features and at least a portion of the category set features; performing image processing based on the image features and the category context features to obtain a target category corresponding to the image to be processed.

[0244] Furthermore, the first processor 21 is also used to perform the aforementioned Figure 2 All or part of the steps in the illustrated embodiments. The electronic device may also include a first communication interface 23 for communication between the electronic device and other devices or communication networks.

[0245] In addition, embodiments of the present invention provide a computer storage medium for storing computer software instructions used by an electronic device, which includes instructions for executing the above-described... Figure 2 The procedure involved in the image classification method in the illustrated embodiment.

[0246] Furthermore, embodiments of the present invention provide a computer program product, comprising: a computer program, which, when executed by a processor of an electronic device, causes the processor to perform... Figure 2 The image classification method shown in the embodiment of the method.

[0247] Figure 14 A schematic diagram of a network model training device provided in this application embodiment; see attached diagram. Figure 14 As shown, this embodiment provides a network model training device that can perform the above-described... Figure 11 The training method for the network model shown, specifically, the training device may include:

[0248] The second acquisition module 31 is used to acquire training images, standard categories corresponding to the training images, and a category set for implementing image classification operations. The category set includes multiple reference categories.

[0249] The second determining module 32 is used to determine the image features of the training images and the category set features corresponding to the category set;

[0250] The second determining module 32 is used to determine the category context features corresponding to the training image based on image features and category set features. The category context features include: image features and at least a portion of the category set features.

[0251] The second processing module 33 is used to perform model training operations based on category context features, training images, and standard categories to obtain a network model for image processing operations.

[0252] Figure 14 The training device for the network model shown can perform... Figure 11 For the methods shown in the embodiments, the parts not described in detail in this embodiment can be referred to the following: Figure 11 The relevant descriptions of the illustrated embodiments are provided below. For the execution process and technical effects of this technical solution, please refer to [link / reference]. Figure 11 The descriptions in the illustrated embodiments will not be repeated here.

[0253] In one possible design, Figure 14 The training device for the network model shown can be implemented as an electronic device. (See attached document.) Figure 15 As shown, the network model training device in this embodiment can be implemented as an electronic device. Specifically, the electronic device may include a second processor 41 and a second memory 42. The second memory 42 is used to store data executed by the corresponding electronic device. Figure 11 In the illustrated embodiment, the program for training the network model is provided, and the second processor 41 is configured to execute the program stored in the second memory 42.

[0254] The program includes one or more computer instructions, wherein when executed by the second processor 41, the one or more computer instructions can perform the following steps: acquiring a training image, a standard category corresponding to the training image, and a category set for implementing image classification operations, the category set including multiple reference categories; determining the image features of the training image and the category set features corresponding to the category set; determining the category context features corresponding to the training image based on the image features and the category set features, the category context features including: image features and at least a portion of the category set features; and performing model training operations based on the category context features, the training image, and the standard category to obtain a network model for implementing image processing operations.

[0255] The second processor 41 is also used to perform the aforementioned Figure 11 All or part of the steps in the illustrated embodiments. The electronic device may also include a second communication interface 43 for communication between the electronic device and other devices or communication networks.

[0256] This invention provides a computer storage medium for storing computer software instructions used by an electronic device, which includes instructions for executing the above-described... Figure 11 The procedure involved in the training method of the network model in the method embodiment shown.

[0257] This invention provides a computer program product, including: a computer program that, when executed by a processor of an electronic device, causes the processor to perform... Figure 11 The training method of the network model in the illustrated embodiment.

[0258] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0259] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0260] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of a necessary general-purpose hardware platform, or by a combination of hardware and software. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a computer product. This application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0261] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable device, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0262] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable device to cause a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0263] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, a network interface, and memory. Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0264] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store data using any method or technology. Data can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store data accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0265] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. An image classification method, characterized in that, include: Obtain the image to be processed and a set of categories for analyzing and processing the image to be processed, the set of categories including multiple reference categories; Determine the image features of the image to be processed and the category set features corresponding to the category set; The image features are projected onto a preset embedding space to obtain the projected features; Determine the association category parameters corresponding to each non-first image processing stage, wherein the association category parameters are different for different image processing stages, and the association category parameters are parameters used to limit the number of association categories; Based on the associated category parameters, associated category features corresponding to the image to be processed in the non-first image processing stage are determined from the category set features. The cosine similarity between the associated category features and the projected features is greater than the cosine similarity between the features of other reference categories and the projected features. The image classification operation for the image to be processed includes multiple image processing stages, including the first image processing stage and the non-first image processing stage. Based on the associated category features and the projected features, category context features corresponding to the image to be processed are determined. The category context features include: the image features and at least a portion of the category set features. Based on the image features and the category context features, a weight offset is determined for adjusting the convolution weights in the network model, which is used to perform image classification operations. The network model and the weight shift are used to process the image features and the category context features to obtain the target category of the image to be processed; Specifically, based on the associated category parameters, determining the associated category features corresponding to the image to be processed in the non-first image processing stage from the category set features includes: Determine the similarity between each reference category in the category set corresponding to the previous image processing stage and the image to be processed; In the category set corresponding to the previous image processing stage, the associated category that satisfies the associated category parameter is selected based on the similarity. In the category set features, all category features corresponding to the associated category are determined as associated category features corresponding to the image to be processed in the non-first image processing stage.

2. The method according to claim 1, characterized in that, Based on the associated category features and the projected features, determine the category context features corresponding to the image to be processed, including: The associated category features and the projected features are integrated to obtain the integrated features; The integrated features are subjected to nonlinear processing and pooling to obtain category context features corresponding to the image to be processed.

3. The method according to claim 1, characterized in that, Based on the image features and the category context features, a weight offset for adjusting the convolution weights in the network model is determined, including: The image features and the category context features are merged to obtain merged features; The merged features are subjected to nonlinear processing to obtain the weight offset.

4. The method according to claim 3, characterized in that, The weight offset is obtained by performing nonlinear processing on the merged features, including: The merged features are subjected to full-connection processing to obtain the processed information; The processed information is normalized and linearly rectified to obtain rectified information; The rectified information is processed by a fully connected layer to obtain the weight offset.

5. The method according to claim 1, characterized in that, The network model and the weight shifts are used to process the image features and the category context features to obtain the target category of the image to be processed, including: Obtain the preset convolution weights corresponding to the network model; Context weights are obtained based on the weight offset and the preset convolution weights; The image features and the category context features are processed using a network model corresponding to the context weights to obtain the target category of the image to be processed.

6. The method according to claim 1, characterized in that, The network model and the weight shifts are used to process the image features and the category context features to obtain the target category of the image to be processed, including: Based on the image features and the category context features, a sampling offset is determined for adjusting the sampling grid in the network model; The image features and category context features are processed using the network model, the weight offset, and the sampling offset to obtain the target category of the image to be processed.

7. An electronic device, characterized in that, include: A memory and a processor; wherein the memory is used to store one or more computer instructions, wherein the one or more computer instructions, when executed by the processor, implement the method as described in any one of claims 1-6.

8. A computer storage medium, characterized in that, Used to store a computer program, which, when executed by a computer, implements the method described in any one of claims 1-6.

Images